2025-01-27 Mon 00:50:50:190.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: http-server-port 8091 2025-01-27 Mon 00:50:50:190.641 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Max-tasks 16 2025-01-27 Mon 00:50:50:190.650 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Max-slice 100 2025-01-27 Mon 00:50:50:190.656 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Collectors 10.0.0.37:8086 2025-01-27 Mon 00:50:50:190.661 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Collectors 10.0.0.32:8086 2025-01-27 Mon 00:50:50:190.666 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Collectors 10.0.0.22:8086 2025-01-27 Mon 00:50:50:190.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: ROLE : Generator 2025-01-27 Mon 00:50:50:190.693 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-01-27 Mon 00:50:50:190.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2660-1. 2025-01-27 Mon 00:50:50:190.703 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: NODE TYPE : Database 2025-01-27 Mon 00:50:50:190.709 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: INSTANCE ID : 0 2025-01-27 Mon 00:50:50:190.714 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-01-27 Mon 00:50:50:190.719 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-01-27 Mon 00:50:50:191.259 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: HTTP Introspect Init 2025-01-27 Mon 00:50:50:191.357 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-01-27 Mon 00:50:50:191.414 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Sandesh Http Server Port: 8091 2025-01-27 Mon 00:50:50:191.450 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-01-27 Mon 00:50:50:191.459 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-01-27 Mon 00:50:50:191.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Idle 2025-01-27 Mon 00:50:50:191.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: No Client: 1737939050191607 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:191.791 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-01-27 Mon 00:50:50:191.804 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-01-27 Mon 00:50:50:191.816 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Cassandra Servers: 10.0.0.37:9042 10.0.0.32:9042 10.0.0.22:9042 2025-01-27 Mon 00:50:50:194.019 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvStart in state Idle 2025-01-27 Mon 00:50:50:194.107 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Disconnect 2025-01-27 Mon 00:50:50:194.216 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-01-27 Mon 00:50:50:194.513 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 00:50:50:194.618 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:194.648 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737939050194095 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:50:50:194.661 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:50:50:194.674 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:194.807 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737939050194201 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:194.830 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:50:50:194.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:194.967 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737939050194291 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:194.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:50:50:194.999 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:195.023 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737939050194321 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:50:50:195.033 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:50:50:195.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:195.170 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737939050194601 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:195.188 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:50:50:209.358 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:50:50:209.399 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:50:50:209.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:50:50:209.516 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:209.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SANDESH: Send FAILED: 1737939050209436 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:50:50:209.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:50:55:215.597 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:50:55:215.635 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:50:55:215.668 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:00:221.959 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:51:00:221.997 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:00:222.018 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:05:223.559 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:51:05:223.600 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:05:223.622 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:10:354.289 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:10:354.345 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:10:354.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:15:380.742 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:15:380.788 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:15:380.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:20:418.716 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:20:418.761 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:20:418.783 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:25:464.586 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:25:464.634 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:25:464.656 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:30:492.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:30:492.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:30:492.419 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:35:518.566 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:35:518.610 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:35:518.632 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:40:570.560 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:40:570.602 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:40:570.625 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:45:629.538 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:45:629.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:50:684.922 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-01-27 Mon 00:51:55:691.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:51:55:691.891 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 00:51:55:691.925 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 00:51:55:691.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a54620 2025-01-27 Mon 00:51:55:691.962 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:691.976 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:691.988 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a54cd0 2025-01-27 Mon 00:51:55:691.993 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.002 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.011 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a4e070 2025-01-27 Mon 00:51:55:692.016 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.027 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.036 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a550d0 2025-01-27 Mon 00:51:55:692.041 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.049 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.058 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a55840 2025-01-27 Mon 00:51:55:692.063 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.071 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.080 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a58fc0 2025-01-27 Mon 00:51:55:692.085 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.093 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.102 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a56180 2025-01-27 Mon 00:51:55:692.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.116 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.124 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a59900 2025-01-27 Mon 00:51:55:692.129 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.137 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.146 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a58680 2025-01-27 Mon 00:51:55:692.154 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.163 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.171 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5a240 2025-01-27 Mon 00:51:55:692.176 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:691.907 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115691759 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:51:55:692.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:692.216 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 00:51:55:692.224 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 00:51:55:692.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5ab80 2025-01-27 Mon 00:51:55:692.481 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.500 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5b4c0 2025-01-27 Mon 00:51:55:692.508 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.516 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.525 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a57d40 2025-01-27 Mon 00:51:55:692.530 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.538 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a57400 2025-01-27 Mon 00:51:55:692.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.567 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.575 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a56ac0 2025-01-27 Mon 00:51:55:692.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:692.592 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:692.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:692.833 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:692.876 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:692.892 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:692.902 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:692.930 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:692.944 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:692.953 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:692.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:692.974 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:692.986 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:692.996 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.014 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.030 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.055 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.090 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.103 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.120 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.134 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.143 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.175 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.193 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.202 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.219 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.243 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.255 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.267 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.289 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.303 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.318 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.330 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.352 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.364 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.373 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.385 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.398 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:693.419 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:693.434 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.466 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.479 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:693.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:692.502 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 00:51:55:693.976 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Idle 2025-01-27 Mon 00:51:55:694.082 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115692450 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 47, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:55:694.208 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:694.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.242 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115692479 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:51:55:694.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:694.259 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.277 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115692867 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 00:51:55:694.284 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:694.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.310 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115692924 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:55:694.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:694.325 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.341 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115693165 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:55:694.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:694.358 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.373 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115693283 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:55:694.379 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:694.387 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 00:51:55:694.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.524 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115694070 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:55:694.537 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:694.625 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:694.837 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115694564 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737939115693972 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737939115694424 ] max_sm_queue_count = 8 ] 2025-01-27 Mon 00:51:55:694.864 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:694.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:695.070 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737939115694613 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:55:695.090 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:694.172 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 00:51:56:694.389 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 00:51:56:694.495 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:51:56:694.524 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737939116694244 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:56:694.533 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:56:694.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:51:56:694.653 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737939116694481 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:56:694.731 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:696.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:34230::10.0.0.32:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 00:51:56:696.716 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:34230::10.0.0.32:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 00:51:56:696.725 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:34230::10.0.0.32:8086(-1):EvTcpConnected 2025-01-27 Mon 00:51:56:696.776 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 00:51:56:696.790 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 00:51:56:696.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit 2025-01-27 Mon 00:51:56:696.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 1 2025-01-27 Mon 00:51:56:696.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 00:51:56:696.915 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 00:51:56:696.936 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737939116696812 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:56:696.943 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 00:51:56:696.952 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 00:51:56:696.959 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 00:51:56:696.969 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 00:51:56:696.976 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:697.082 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737939116696897 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:56:697.099 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:698.779 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 00:51:56:698.820 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 00:51:56:698.830 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 00:51:56:698.852 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 00:51:56:699.003 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737939115693972 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737939115694424 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 8 proxy seq 10 2025-01-27 Mon 00:51:56:699.029 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 00:51:56:699.053 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-01-27 Mon 00:51:56:699.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 00:51:56:699.071 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-01-27 Mon 00:51:56:699.078 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 00:51:56:699.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Established 2025-01-27 Mon 00:52:06:643.044 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: OnMessage control in state: Established 2025-01-27 Mon 00:52:06:643.102 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-01-27 Mon 00:52:06:643.112 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Established : EvSandeshMessageRecv 2025-01-27 Mon 00:52:06:643.162 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Received Ctrl Message with size 3 2025-01-27 Mon 00:52:06:643.176 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-01-27 Mon 00:52:06:643.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-01-27 Mon 00:52:06:643.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-01-27 Mon 00:52:06:643.206 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 00:52:25:694.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-01-27 Mon 00:53:25:695.335 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-01-27 Mon 00:54:25:695.874 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-01-27 Mon 00:54:54:727.435 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:50768 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-01-27 Mon 00:55:18:934.030 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:49952 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-01-27 Mon 00:55:25:699.430 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-01-27 Mon 00:56:25:700.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-01-27 Mon 00:57:25:702.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-01-27 Mon 00:58:25:703.937 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-01-27 Mon 00:59:25:704.458 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-01-27 Mon 01:00:25:705.643 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-01-27 Mon 01:01:25:708.357 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-01-27 Mon 01:02:25:708.835 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-01-27 Mon 01:03:25:709.888 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-01-27 Mon 01:04:25:712.552 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-01-27 Mon 01:05:25:713.096 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-01-27 Mon 01:06:25:713.629 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-01-27 Mon 01:07:25:714.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-01-27 Mon 01:08:25:715.848 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-01-27 Mon 01:09:25:716.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-01-27 Mon 01:10:25:718.733 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-01-27 Mon 01:11:25:719.389 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-01-27 Mon 01:12:25:719.895 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-01-27 Mon 01:13:25:720.928 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-01-27 Mon 01:14:25:721.390 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-01-27 Mon 01:15:25:722.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-01-27 Mon 01:16:25:722.941 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-01-27 Mon 01:17:25:723.450 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-01-27 Mon 01:18:25:723.919 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-01-27 Mon 01:19:25:724.490 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-01-27 Mon 01:20:25:724.949 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-01-27 Mon 01:21:25:725.947 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-01-27 Mon 01:22:25:727.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-01-27 Mon 01:23:25:727.702 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-01-27 Mon 01:24:25:728.807 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-01-27 Mon 01:25:25:729.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-01-27 Mon 01:26:25:729.737 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-01-27 Mon 01:27:25:730.188 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-01-27 Mon 01:28:25:730.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-01-27 Mon 01:29:25:731.320 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-01-27 Mon 01:30:25:731.769 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-01-27 Mon 01:31:25:732.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-01-27 Mon 01:32:25:732.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-01-27 Mon 01:33:25:733.330 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-01-27 Mon 01:34:25:733.828 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-01-27 Mon 01:35:25:734.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-01-27 Mon 01:36:25:735.738 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-01-27 Mon 01:37:25:737.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-01-27 Mon 01:38:25:737.614 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-01-27 Mon 01:39:25:738.195 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-01-27 Mon 01:40:25:738.725 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-01-27 Mon 01:41:25:739.286 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-01-27 Mon 01:42:25:739.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-01-27 Mon 01:43:25:740.312 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-01-27 Mon 01:44:25:740.822 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094702860032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-01-27 Mon 01:45:25:741.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-01-27 Mon 01:46:25:741.801 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-01-27 Mon 01:47:25:742.605 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-01-27 Mon 01:48:25:745.467 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-01-27 Mon 01:49:25:745.934 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-01-27 Mon 01:50:25:746.482 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-01-27 Mon 01:51:25:746.920 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-01-27 Mon 01:52:25:747.380 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-01-27 Mon 01:53:25:748.403 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-01-27 Mon 01:54:25:748.936 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-01-27 Mon 01:55:25:749.466 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-01-27 Mon 01:56:25:750.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-01-27 Mon 01:57:25:750.532 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-01-27 Mon 01:58:25:751.062 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-01-27 Mon 01:59:25:751.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 68 without seqno , total = 0 2025-01-27 Mon 02:00:25:752.140 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 69 without seqno , total = 0 2025-01-27 Mon 02:01:25:752.703 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 70 without seqno , total = 0 2025-01-27 Mon 02:02:25:753.196 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 71 without seqno , total = 0 2025-01-27 Mon 02:03:25:753.805 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 72 without seqno , total = 0 2025-01-27 Mon 02:04:25:754.245 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 73 without seqno , total = 0 2025-01-27 Mon 02:05:25:757.760 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 74 without seqno , total = 0 2025-01-27 Mon 02:06:25:758.310 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 75 without seqno , total = 0 2025-01-27 Mon 02:06:36:955.874 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a56ac0 2025-01-27 Mon 02:06:36:955.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:36:955.993 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a57400 2025-01-27 Mon 02:06:36:956.005 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:36:956.129 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a57d40 2025-01-27 Mon 02:06:36:956.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:36:956.163 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a58680 2025-01-27 Mon 02:06:36:956.173 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:36:956.196 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:06:36:956.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a56180 2025-01-27 Mon 02:06:36:956.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:36:956.593 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:06:36:956.653 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:06:36:956.669 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:06:36:956.682 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:06:36:956.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:06:37:021.579 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a5b4c0 2025-01-27 Mon 02:06:37:021.640 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:37:021.700 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a5ab80 2025-01-27 Mon 02:06:37:021.712 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:37:021.731 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a5a240 2025-01-27 Mon 02:06:37:021.741 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:37:021.757 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a59900 2025-01-27 Mon 02:06:37:021.766 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:37:021.788 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:06:37:021.796 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a58fc0 2025-01-27 Mon 02:06:37:021.804 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:06:37:022.198 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:06:37:022.224 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:06:37:022.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:06:37:022.263 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:06:37:022.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:06:41:956.696 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:06:41:956.837 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:06:41:956.873 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:06:41:956.898 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:06:41:956.922 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:06:41:957.422 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:06:41:957.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:06:41:957.474 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:06:41:957.488 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:06:41:957.510 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:06:42:022.273 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:06:42:022.407 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:06:42:022.448 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:06:42:022.474 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:06:42:022.497 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:06:42:022.769 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:06:42:022.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:06:42:022.828 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:06:42:022.848 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:06:42:022.861 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:06:46:945.014 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:34230::10.0.0.32:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-01-27 Mon 02:06:46:945.053 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:34230::10.0.0.32:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:06:46:945.073 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:34230::10.0.0.32:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:06:46:945.081 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:34230::10.0.0.32:8086(-1):EvTcpClose 2025-01-27 Mon 02:06:46:945.296 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpClose in state Established 2025-01-27 Mon 02:06:46:945.325 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Established : EvTcpClose 2025-01-27 Mon 02:06:46:945.626 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:06:46:945.747 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:46:945.994 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943606945436 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = cn-jenkins-deploy-platform-ansible-os-2660-2. collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3623290 calls = 380 average_bytes = 9534 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 115, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 381, ] ] ] 2025-01-27 Mon 02:06:46:946.024 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:46:946.038 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:46:946.068 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943606945475 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:46:946.077 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:46:946.090 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-01-27 Mon 02:06:46:946.114 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:46:946.312 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943606945725 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3623290 calls = 380 average_bytes = 9534 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 115, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 381, ] ] ] 2025-01-27 Mon 02:06:46:946.332 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:46:952.738 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:43602::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:06:46:952.771 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:43602::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:06:46:952.782 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:43602::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:06:46:956.861 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:06:46:956.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:06:46:957.026 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: ClientInit 2025-01-27 Mon 02:06:46:957.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 2 2025-01-27 Mon 02:06:46:957.179 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:46:957.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:46:957.214 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SANDESH: Send FAILED: 1737943606956953 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:46:957.228 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:46:957.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:46:957.244 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:06:46:957.255 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:46:957.262 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:06:46:957.392 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SANDESH: Send FAILED: 1737943606957168 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3623290 calls = 380 average_bytes = 9534 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 116, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 381, ] ] ] 2025-01-27 Mon 02:06:46:957.409 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:06:46:958.408 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:06:46:958.487 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:06:46:958.521 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:06:46:958.554 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:06:46:958.602 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:06:46:958.722 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:06:46:958.759 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:06:46:958.773 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:06:46:958.803 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:06:46:958.866 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:06:47:022.833 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:06:47:022.945 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:06:47:022.980 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:06:47:023.028 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:06:47:023.062 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:06:47:023.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:06:47:023.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:06:47:023.238 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:06:47:023.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:06:47:023.340 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:06:47:058.347 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:43602::10.0.0.22:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-01-27 Mon 02:06:47:058.387 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:43602::10.0.0.22:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:06:47:058.422 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:43602::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:06:47:058.435 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:43602::10.0.0.22:8086(-1):EvTcpClose 2025-01-27 Mon 02:06:47:060.075 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:06:47:060.109 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:06:47:060.242 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Return to idle with 1000 2025-01-27 Mon 02:06:47:060.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Idle 2025-01-27 Mon 02:06:47:060.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:47:060.471 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943607060198 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3624863 calls = 381 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 118, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 382, ] ] ] 2025-01-27 Mon 02:06:47:060.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:47:060.502 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:47:060.530 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943607060228 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:47:060.544 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:47:060.558 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:06:47:060.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:47:060.735 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943607060304 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3624863 calls = 381 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 118, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 382, ] ] ] 2025-01-27 Mon 02:06:47:060.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:48:060.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:06:48:060.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:06:48:060.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:48:061.006 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943608060692 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:48:061.015 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:48:061.024 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:48:061.167 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943608060966 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3624863 calls = 381 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 118, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 382, ] ] ] 2025-01-27 Mon 02:06:48:061.188 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:48:063.141 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:60126::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:06:48:063.162 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:60126::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:06:48:063.170 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:60126::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:06:48:063.375 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:06:48:063.387 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:06:48:063.417 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: ClientInit 2025-01-27 Mon 02:06:48:063.431 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 3 2025-01-27 Mon 02:06:48:063.512 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:48:063.521 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:48:063.554 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SANDESH: Send FAILED: 1737943608063411 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:48:063.570 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:48:063.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:48:063.587 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:06:48:063.598 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:48:063.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:06:48:063.736 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: SANDESH: Send FAILED: 1737943608063502 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3624863 calls = 381 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 119, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 382, ] ] ] 2025-01-27 Mon 02:06:48:063.754 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094711256832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:06:48:070.295 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:06:48:070.334 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:06:48:070.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:06:48:070.370 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:06:48:070.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3624863 calls = 381 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 397 dequeues = 397 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 405 dequeues = 405 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1737939116699107 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1737943560737355 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 377 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 3594971 calls = 376 average_bytes = 9561 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 119, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 382, ] ] max_sm_queue_count = 8 proxy seq 316 2025-01-27 Mon 02:06:48:070.611 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:06:48:070.634 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 20 2025-01-27 Mon 02:06:48:070.648 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:06:48:070.679 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 75 2025-01-27 Mon 02:06:48:070.689 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:06:48:070.697 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:06:48:070.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Established 2025-01-27 Mon 02:06:51:958.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:06:51:958.916 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:06:51:958.955 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:06:51:958.983 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:06:51:959.009 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:06:51:959.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a56ac0 2025-01-27 Mon 02:06:51:959.404 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:51:959.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:51:959.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a57400 2025-01-27 Mon 02:06:51:959.472 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:51:959.485 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:51:959.498 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a57d40 2025-01-27 Mon 02:06:51:959.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:51:959.520 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:51:959.530 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a58680 2025-01-27 Mon 02:06:51:959.549 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:51:959.557 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:51:959.611 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a56180 2025-01-27 Mon 02:06:51:959.618 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:51:959.627 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:51:959.825 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:51:959.861 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:51:959.890 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:51:960.049 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:51:960.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:51:960.332 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:51:960.359 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:51:960.372 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:51:960.386 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:51:960.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:51:960.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:51:960.415 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:51:960.460 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:51:960.476 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:51:960.680 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:52:024.178 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:06:52:024.283 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:06:52:024.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:06:52:024.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:06:52:024.371 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:06:52:024.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5b4c0 2025-01-27 Mon 02:06:52:024.589 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:52:024.642 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:52:024.667 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5ab80 2025-01-27 Mon 02:06:52:024.673 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:52:024.682 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:52:024.709 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5a240 2025-01-27 Mon 02:06:52:024.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:52:024.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:52:024.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a59900 2025-01-27 Mon 02:06:52:024.757 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:52:024.765 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:52:024.774 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a58fc0 2025-01-27 Mon 02:06:52:024.779 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:52:024.787 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:52:024.986 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:52:025.037 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:52:025.062 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:52:025.086 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:52:025.114 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:52:025.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:52:025.145 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:52:025.200 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:52:025.216 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:52:025.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:52:025.243 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:52:025.257 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:52:025.269 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:52:025.289 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:52:025.305 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:07:25:759.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 76 without seqno , total = 0 2025-01-27 Mon 02:08:25:760.437 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 77 without seqno , total = 0 2025-01-27 Mon 02:09:21:869.788 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a55840 2025-01-27 Mon 02:09:21:869.871 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:869.930 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a550d0 2025-01-27 Mon 02:09:21:869.944 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:869.969 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a4e070 2025-01-27 Mon 02:09:21:869.980 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:870.003 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:21:870.013 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:870.041 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:09:21:870.049 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a54620 2025-01-27 Mon 02:09:21:870.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:883.084 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:21:883.128 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:21:883.144 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:21:883.159 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:21:883.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:60126::10.0.0.37:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:09:21:883.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:60126::10.0.0.37:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:21:883.219 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:60126::10.0.0.37:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:21:884.094 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpClose in state Established 2025-01-27 Mon 02:09:21:885.568 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Established : EvTcpClose 2025-01-27 Mon 02:09:21:885.848 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:09:21:885.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:886.118 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943761885691 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = cn-jenkins-deploy-platform-ansible-os-2660-1. collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3804097 calls = 401 average_bytes = 9486 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 79 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 312 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 405, ] ] ] 2025-01-27 Mon 02:09:21:886.140 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:886.151 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:886.174 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943761885734 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:09:21:886.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:21:886.190 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-01-27 Mon 02:09:21:886.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:886.434 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943761885946 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3804097 calls = 401 average_bytes = 9486 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 79 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 312 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 405, ] ] ] 2025-01-27 Mon 02:09:21:886.452 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:884.126 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:21:888.517 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:37118::10.0.0.32:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:21:888.535 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:37118::10.0.0.32:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:21:888.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:37118::10.0.0.32:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:21:892.667 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:21:892.691 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:21:892.734 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: ClientInit 2025-01-27 Mon 02:09:21:892.752 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 4 2025-01-27 Mon 02:09:21:892.840 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:892.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:21:892.874 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SANDESH: Send FAILED: 1737943761892726 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:09:21:892.882 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:21:892.891 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:892.899 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:21:892.911 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:892.918 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:893.054 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SANDESH: Send FAILED: 1737943761892830 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3804097 calls = 401 average_bytes = 9486 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 79 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 312 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 405, ] ] ] 2025-01-27 Mon 02:09:21:893.071 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:945.272 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:09:21:945.332 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:09:21:945.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:09:21:945.370 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:09:21:945.603 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3804097 calls = 401 average_bytes = 9486 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 433 dequeues = 433 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 449 dequeues = 449 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1737943608070723 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1737943705760497 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 19 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 153541 calls = 16 average_bytes = 9596 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 79 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 312 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 405, ] ] max_sm_queue_count = 8 proxy seq 330 2025-01-27 Mon 02:09:21:945.635 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:09:21:945.661 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 27 2025-01-27 Mon 02:09:21:945.676 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:09:21:945.717 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 78 2025-01-27 Mon 02:09:21:945.728 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:09:21:945.736 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:09:21:945.769 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Established 2025-01-27 Mon 02:09:21:957.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a57d40 2025-01-27 Mon 02:09:21:957.257 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:957.372 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a58680 2025-01-27 Mon 02:09:21:957.386 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:957.413 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a57400 2025-01-27 Mon 02:09:21:957.423 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:957.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:21:957.452 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:957.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:09:21:957.482 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a56180 2025-01-27 Mon 02:09:21:957.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:959.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:21:960.130 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:21:960.154 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:21:960.168 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:21:960.182 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:21:966.668 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:37118::10.0.0.32:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:21:966.697 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:37118::10.0.0.32:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:21:967.898 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvTcpClose in state Established 2025-01-27 Mon 02:09:21:967.918 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Established : EvTcpClose 2025-01-27 Mon 02:09:21:968.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:09:21:968.311 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:968.488 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943761968023 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = cn-jenkins-deploy-platform-ansible-os-2660-2. collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3857801 calls = 406 average_bytes = 9501 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 160 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 413, ] ] ] 2025-01-27 Mon 02:09:21:968.510 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:968.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:968.545 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943761968060 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down.10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:09:21:968.559 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:21:968.568 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-01-27 Mon 02:09:21:968.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:968.703 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943761968297 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3857801 calls = 406 average_bytes = 9501 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 160 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 413, ] ] ] 2025-01-27 Mon 02:09:21:968.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:974.944 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:56816::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:21:974.978 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:56816::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:21:974.989 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:56816::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:21:976.496 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:21:976.515 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:21:976.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: ClientInit 2025-01-27 Mon 02:09:21:976.575 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 5 2025-01-27 Mon 02:09:21:976.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:976.671 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:21:976.694 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943761976543 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down.10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:09:21:976.702 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:21:976.711 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:976.717 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:21:976.728 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:976.734 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:976.866 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943761976652 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3857801 calls = 406 average_bytes = 9501 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 160 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 413, ] ] ] 2025-01-27 Mon 02:09:21:976.883 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:037.354 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:09:22:037.392 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:22:037.405 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a59900 2025-01-27 Mon 02:09:22:037.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.575 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a5a240 2025-01-27 Mon 02:09:22:037.586 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.605 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:22:037.614 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.634 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Disconnect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:22:037.643 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:038.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:22:038.376 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:22:038.402 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943762037686 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:22:038.417 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:22:039.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:22:039.458 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:22:039.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:22:039.498 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:22:039.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:22:181.968 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:56816::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:22:182.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:56816::10.0.0.22:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:22:190.049 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:09:22:190.087 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:09:22:190.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Return to idle with 1000 2025-01-27 Mon 02:09:22:190.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Idle 2025-01-27 Mon 02:09:22:190.279 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:22:190.420 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943762190166 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3859374 calls = 407 average_bytes = 9482 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 414, ] ] ] 2025-01-27 Mon 02:09:22:190.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:190.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:22:190.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943762190199 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:22:190.483 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:22:190.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:22:190.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:22:190.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943762190270 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3859374 calls = 407 average_bytes = 9482 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 414, ] ] ] 2025-01-27 Mon 02:09:22:190.668 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:23:190.645 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:23:190.807 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:09:23:190.900 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:23:190.927 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943763190698 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:23:190.937 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:23:190.945 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:23:191.079 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943763190890 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3859374 calls = 407 average_bytes = 9482 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 414, ] ] ] 2025-01-27 Mon 02:09:23:191.100 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:23:195.435 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:45470::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:23:195.459 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:45470::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:23:195.467 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:45470::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:23:196.038 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:23:196.052 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:23:196.083 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: ClientInit 2025-01-27 Mon 02:09:23:196.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 6 2025-01-27 Mon 02:09:23:196.175 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:23:196.184 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:23:196.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943763196076 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:23:196.220 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:23:196.228 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:23:196.235 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:23:196.246 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:23:196.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:23:196.379 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943763196166 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3859374 calls = 407 average_bytes = 9482 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 414, ] ] ] 2025-01-27 Mon 02:09:23:196.393 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:23:198.353 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:45470::10.0.0.37:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:23:198.370 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:45470::10.0.0.37:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:23:198.894 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:09:23:198.911 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:09:23:199.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Return to idle with 1000 2025-01-27 Mon 02:09:23:199.031 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Idle 2025-01-27 Mon 02:09:23:199.100 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:23:199.251 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943763198976 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3860947 calls = 408 average_bytes = 9463 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-01-27 Mon 02:09:23:199.268 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:23:199.278 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:23:199.299 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943763198997 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:23:199.309 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:23:199.318 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:23:199.327 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:23:199.450 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: SANDESH: Send FAILED: 1737943763199088 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3860947 calls = 408 average_bytes = 9463 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-01-27 Mon 02:09:23:199.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094686070528, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:199.740 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:24:200.025 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:09:24:200.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:24:200.166 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943764199838 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:24:200.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:24:200.196 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:24:200.392 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943764200118 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3860947 calls = 408 average_bytes = 9463 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-01-27 Mon 02:09:24:200.411 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:202.439 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:46718::10.0.0.32:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:24:202.470 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:46718::10.0.0.32:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:24:202.478 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:46718::10.0.0.32:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:24:202.791 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:24:202.806 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:24:202.844 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: ClientInit 2025-01-27 Mon 02:09:24:202.858 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 7 2025-01-27 Mon 02:09:24:202.945 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:24:202.954 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:24:202.978 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943764202837 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:24:202.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:24:202.994 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:24:203.001 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:24:203.013 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:24:203.019 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:203.151 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943764202936 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3860947 calls = 408 average_bytes = 9463 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-01-27 Mon 02:09:24:203.170 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:204.363 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:46718::10.0.0.32:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:24:204.389 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:46718::10.0.0.32:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:24:204.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:09:24:204.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:09:24:204.987 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Return to idle with 1000 2025-01-27 Mon 02:09:24:205.021 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Idle 2025-01-27 Mon 02:09:24:205.109 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:24:205.273 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943764204885 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3862520 calls = 409 average_bytes = 9443 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-01-27 Mon 02:09:24:205.302 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:205.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:24:205.337 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943764204909 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:24:205.344 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:24:205.355 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:24:205.390 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:24:205.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943764205092 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3862520 calls = 409 average_bytes = 9443 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-01-27 Mon 02:09:24:205.535 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:207.249 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:25:207.463 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:09:25:207.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:25:207.595 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943765207326 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:25:207.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:25:207.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:25:207.747 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943765207545 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3862520 calls = 409 average_bytes = 9443 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-01-27 Mon 02:09:25:207.772 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:212.467 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:56718::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:25:212.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:56718::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:25:212.516 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:56718::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:25:212.612 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:25:212.645 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:25:212.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: ClientInit 2025-01-27 Mon 02:09:25:212.705 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 8 2025-01-27 Mon 02:09:25:212.800 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:25:212.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:25:212.834 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SANDESH: Send FAILED: 1737943765212677 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:25:212.857 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:25:212.867 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:25:212.873 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:25:212.884 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:25:212.890 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:213.030 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: SANDESH: Send FAILED: 1737943765212788 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3862520 calls = 409 average_bytes = 9443 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-01-27 Mon 02:09:25:213.046 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093910148864, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:762.197 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 78 without seqno , total = 0 2025-01-27 Mon 02:09:25:762.257 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:25:762.268 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:762.352 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943765762108 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 481 dequeues = 481 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 516 dequeues = 516 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1737943765212685 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1737943765212883 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 8 ] 2025-01-27 Mon 02:09:25:762.365 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:762.375 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:25:762.382 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:25:762.506 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943765762181 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3864093 calls = 410 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-01-27 Mon 02:09:25:762.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:26:883.204 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:26:883.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:26:883.503 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:26:883.586 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:26:883.627 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:26:883.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:26:883.661 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:26:883.711 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:26:886.564 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:26:886.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:26:960.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:26:960.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:26:960.295 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:26:960.331 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:26:960.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:26:960.574 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:26:960.610 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:26:960.647 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:26:960.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:27:039.481 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:27:039.649 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:27:039.707 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:27:039.750 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:27:039.791 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:27:039.853 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:27:039.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:27:039.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:27:039.927 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:27:039.947 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:27:222.044 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:31:883.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:31:883.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:31:883.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:31:883.886 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:31:883.917 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:31:883.952 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:31:883.968 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:31:883.980 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:31:886.726 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:31:886.814 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:31:960.639 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:31:960.800 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:31:960.844 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:31:960.992 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:31:961.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:31:961.362 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:31:961.386 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:31:961.404 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:32:039.984 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:32:040.126 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:32:040.158 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:32:040.184 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:32:040.208 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:32:040.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:32:040.528 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:32:040.546 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:32:040.804 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:32:040.824 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:32:222.148 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:32:222.588 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:33:213.264 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: 10.0.0.22:8086 EvConnectTimerExpired in state ClientInit 2025-01-27 Mon 02:09:33:213.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-01-27 Mon 02:09:33:213.336 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: ClientInit : EvConnectTimerExpired 2025-01-27 Mon 02:09:33:213.510 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Return to idle with 1000 2025-01-27 Mon 02:09:33:213.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Idle 2025-01-27 Mon 02:09:33:213.596 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:33:213.740 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943773213419 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3864093 calls = 410 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-01-27 Mon 02:09:33:213.767 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:33:213.777 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:33:213.799 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943773213449 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:33:213.806 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:33:213.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:33:213.827 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:33:213.978 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: SANDESH: Send FAILED: 1737943773213586 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3864093 calls = 410 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-01-27 Mon 02:09:33:213.994 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094707058432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:214.084 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:34:214.407 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:09:34:214.519 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:34:214.559 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943774214183 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:34:214.582 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:34:214.592 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:34:214.736 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: SANDESH: Send FAILED: 1737943774214506 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3864093 calls = 410 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-01-27 Mon 02:09:34:214.753 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094950078208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:217.125 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:50442::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:34:217.166 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:50442::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:34:217.175 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:50442::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:34:217.273 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:34:217.320 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:34:217.362 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: ClientInit 2025-01-27 Mon 02:09:34:217.388 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 9 2025-01-27 Mon 02:09:34:217.479 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:34:217.489 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:34:217.513 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SANDESH: Send FAILED: 1737943774217354 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:34:217.520 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:34:217.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:34:217.536 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:34:217.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:34:217.564 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:217.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SANDESH: Send FAILED: 1737943774217468 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3864093 calls = 410 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-01-27 Mon 02:09:34:217.716 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:313.380 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:50442::10.0.0.37:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-01-27 Mon 02:09:34:313.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.37:50442::10.0.0.37:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:09:34:313.469 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:50442::10.0.0.37:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:34:313.477 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:50442::10.0.0.37:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:34:313.575 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:09:34:313.615 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:09:34:313.746 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Return to idle with 1000 2025-01-27 Mon 02:09:34:313.755 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Idle 2025-01-27 Mon 02:09:34:313.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:34:315.776 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943774313697 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:09:34:315.814 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:315.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:34:315.855 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943774313728 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:34:315.862 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:34:315.872 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:34:315.903 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:34:316.031 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SANDESH: Send FAILED: 1737943774313803 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:09:34:316.050 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:35:314.639 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:35:314.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:09:35:314.975 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:35:315.004 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943775314719 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:35:315.013 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:35:315.023 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:35:315.163 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943775314961 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:09:35:315.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:35:315.883 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:09:35:315.892 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:09:35:316.685 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:09:35:316.695 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:09:35:316.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:09:35:316.825 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Idle 2025-01-27 Mon 02:09:35:316.882 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:35:317.014 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SANDESH: Send FAILED: 1737943775316753 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:09:35:317.035 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:35:317.046 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:35:317.067 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SANDESH: Send FAILED: 1737943775316774 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:35:317.073 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:35:317.082 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:35:317.091 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:35:317.214 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: SANDESH: Send FAILED: 1737943775316873 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:09:35:317.229 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094941681408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:36:883.883 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:36:884.047 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:36:884.074 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:36:884.109 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:36:884.138 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:36:884.172 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:36:884.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:36:884.200 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:36:886.859 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:36:886.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:36:961.403 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:36:961.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:36:961.672 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:36:961.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:36:962.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:36:962.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:36:962.133 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:36:962.152 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:37:040.602 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:37:040.802 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:37:040.871 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:37:040.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:37:040.962 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:37:041.121 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:37:041.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:37:041.236 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:37:041.256 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:37:041.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:37:222.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:37:223.010 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:41:884.109 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:41:884.277 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:41:884.301 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:41:884.334 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:41:884.362 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:41:884.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:41:884.411 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:41:884.423 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:41:887.025 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:41:887.141 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:41:962.131 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:41:962.246 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:41:962.279 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:41:962.303 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:41:962.618 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:41:962.680 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:41:962.695 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:41:962.709 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:42:041.243 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:42:041.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:42:041.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:42:041.489 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:42:041.525 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:42:041.656 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:42:041.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:42:041.768 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:42:041.811 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:42:041.830 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:42:223.065 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:42:223.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:46:884.341 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:46:884.500 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:46:884.527 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:46:884.571 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:46:884.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:46:884.647 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:46:884.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:46:884.676 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:46:887.176 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:46:887.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:46:962.710 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:46:962.841 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:46:962.873 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:46:962.898 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:46:963.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:46:963.336 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:46:963.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:46:963.363 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:47:041.771 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:47:041.944 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:47:041.990 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:47:042.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:47:042.069 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:47:042.295 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:47:042.340 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:47:042.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:47:042.388 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:47:042.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:47:223.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:47:224.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:51:884.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:51:884.717 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:51:884.744 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:51:884.777 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:51:884.806 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:51:884.839 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:51:884.854 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:51:884.867 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:51:887.264 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:51:887.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:51:963.380 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:51:963.516 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:51:963.560 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:51:963.591 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:51:963.885 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:51:963.919 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:51:963.935 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:51:963.952 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:52:042.363 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:52:042.490 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:52:042.525 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:52:042.561 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:52:042.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:52:042.738 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:52:042.765 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:52:042.797 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:52:042.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:52:042.831 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:52:224.314 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:52:224.893 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:09:55:763.501 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:55:763.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943795763308 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 499 dequeues = 499 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 543 dequeues = 543 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943775316824 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943775317090 ] max_sm_queue_count = 8 ] 2025-01-27 Mon 02:09:55:763.667 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:55:763.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:55:763.816 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943795763459 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 81, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:09:55:763.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:56:884.811 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:09:56:885.076 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:09:56:885.113 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:09:56:885.153 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:09:56:885.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:09:56:885.220 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:09:56:885.236 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:09:56:885.248 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:09:56:887.364 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:09:56:887.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:09:56:963.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:09:56:964.111 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:09:56:964.156 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:09:56:964.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:09:56:964.525 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:09:56:964.588 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:09:56:964.621 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:09:56:964.634 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:09:57:042.813 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:09:57:042.918 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:09:57:042.961 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:09:57:042.989 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:09:57:043.014 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:09:57:043.139 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:09:57:043.190 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:09:57:043.208 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:09:57:043.223 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:09:57:043.238 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:09:57:224.982 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:09:57:225.559 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:00:746.095 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:00:746.348 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943800746031 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-01-27 Mon 02:10:00:746.387 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-01-27 Mon 02:10:01:885.148 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:01:885.287 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:01:885.311 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:01:885.344 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:01:885.376 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:01:885.426 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:01:885.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:01:885.455 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:01:887.488 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:01:887.600 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:01:964.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:01:964.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:01:964.787 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:01:964.818 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:01:965.153 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:01:965.221 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:01:965.238 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:01:965.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:02:043.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:02:043.371 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:02:043.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:02:043.439 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:02:043.469 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:02:043.700 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:02:043.790 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:02:043.813 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:02:043.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:02:043.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:02:225.636 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:02:226.215 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:05:317.523 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:10:05:317.777 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:10:05:317.915 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:05:317.947 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943805317627 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:05:317.956 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:05:317.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:05:318.100 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943805317902 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 84, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:05:318.119 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:05:318.763 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:10:05:318.775 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:10:05:319.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:10:05:319.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:10:05:319.441 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:10:05:319.470 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Idle 2025-01-27 Mon 02:10:05:319.528 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:05:319.672 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SANDESH: Send FAILED: 1737943805319394 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:05:319.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:05:319.702 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:05:319.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SANDESH: Send FAILED: 1737943805319418 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:05:319.731 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:05:319.739 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:10:05:319.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:05:319.870 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SANDESH: Send FAILED: 1737943805319519 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 188, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:05:319.885 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:06:885.355 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:06:885.495 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:06:885.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:06:885.569 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:06:885.611 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:06:885.647 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:06:885.664 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:06:885.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:06:887.641 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:06:887.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:06:965.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:06:965.384 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:06:965.433 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:06:965.465 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:06:965.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:06:965.741 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:06:965.760 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:06:965.774 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:07:043.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:07:043.936 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:07:043.984 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:07:044.014 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:07:044.043 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:07:044.087 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:07:044.147 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:07:044.164 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:07:044.180 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:07:044.195 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:07:226.310 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:07:226.767 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:11:885.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:11:885.727 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:11:889.124 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:11:889.205 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:11:889.240 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:11:889.282 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:11:889.300 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:11:889.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:11:889.323 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:11:889.355 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:11:965.731 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:11:965.892 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:11:965.930 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:11:965.962 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:11:966.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:11:966.369 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:11:966.385 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:11:966.399 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:12:044.166 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:12:044.261 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:12:044.300 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:12:044.327 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:12:044.356 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:12:044.409 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:12:044.487 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:12:044.503 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:12:044.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:12:044.637 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:12:226.831 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:12:227.296 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:16:889.154 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:16:889.337 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:16:889.370 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:16:889.421 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:16:889.482 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:16:889.506 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:16:889.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:16:889.582 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:16:889.628 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:16:889.645 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:16:966.397 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:16:966.523 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:16:966.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:16:966.592 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:16:966.928 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:16:966.962 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:16:966.976 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:16:966.988 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:17:044.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:17:044.620 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:17:044.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:17:044.692 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:17:044.723 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:17:044.830 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:17:044.856 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:17:044.869 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:17:044.882 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:17:044.895 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:17:227.359 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:17:227.864 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:21:889.408 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:21:889.558 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:21:889.583 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:21:889.618 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:21:889.663 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:21:889.679 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:21:889.689 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:21:889.717 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:21:889.747 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:21:889.761 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:21:967.006 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:21:967.150 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:21:967.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:21:967.241 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:21:967.567 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:21:967.617 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:21:967.633 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:21:967.649 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:22:044.897 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:22:044.990 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:22:045.028 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:22:045.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:22:045.084 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:22:045.137 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:22:045.274 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:22:045.306 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:22:045.330 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:22:045.345 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:22:227.930 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:22:228.358 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:25:763.763 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 79 without seqno , total = 0 2025-01-27 Mon 02:10:25:763.925 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:25:764.038 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943825763616 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 507 dequeues = 507 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 554 dequeues = 554 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943805319469 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943805319746 ] max_sm_queue_count = 8 ] 2025-01-27 Mon 02:10:25:764.072 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:25:764.086 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:25:764.353 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943825763743 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 188, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 89, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:25:764.384 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:26:889.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:26:889.894 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:26:889.931 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:26:889.975 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:26:889.997 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:26:890.010 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:26:890.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:26:890.047 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:26:890.076 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:26:890.090 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:26:967.644 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:26:967.753 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:26:967.786 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:26:967.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:26:968.106 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:26:968.148 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:26:968.172 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:26:968.194 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:27:045.195 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:27:045.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:27:045.378 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:27:045.438 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:27:045.466 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:27:045.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:27:045.744 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:27:045.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:27:045.813 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:27:045.839 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:27:228.445 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:27:229.054 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:31:890.037 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:31:890.177 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:31:890.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:31:890.256 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:31:890.267 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:31:890.294 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:31:890.320 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:31:890.351 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:31:890.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:31:890.378 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:31:968.182 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:31:968.286 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:31:968.320 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:31:968.346 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:31:968.648 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:31:968.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:31:968.928 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:31:968.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:32:045.687 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:32:045.823 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:32:045.879 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:32:045.916 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:32:045.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:32:046.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:32:046.113 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:32:046.135 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:32:046.149 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:32:046.162 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:32:229.121 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:32:229.625 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:35:319.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:10:35:319.792 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:10:35:319.905 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:35:319.933 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SANDESH: Send FAILED: 1737943835319649 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:35:319.942 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:35:319.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:35:320.089 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SANDESH: Send FAILED: 1737943835319891 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 188, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 91, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:35:320.111 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:35:320.897 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:10:35:320.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:10:35:321.593 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:10:35:321.605 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:10:35:321.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:10:35:321.755 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Idle 2025-01-27 Mon 02:10:35:321.813 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:35:321.967 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943835321667 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 188, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 93, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:35:321.987 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:35:321.998 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:35:322.019 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943835321697 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:35:322.026 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:35:322.035 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:10:35:322.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:35:322.173 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SANDESH: Send FAILED: 1737943835321804 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 189, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 93, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:35:322.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:36:891.708 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:36:891.825 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:36:891.859 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:36:891.888 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:36:891.916 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:36:891.954 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:36:891.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:36:891.990 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:36:892.002 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:36:892.014 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:36:968.713 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:36:968.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:36:968.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:36:969.003 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:36:969.145 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:36:969.170 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:36:969.191 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:36:969.238 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:37:050.015 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:37:050.122 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:37:050.159 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:37:050.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:37:050.221 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:37:050.359 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:37:050.387 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:37:050.400 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:37:050.414 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:37:050.426 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:37:229.805 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:37:230.221 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:41:892.017 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:41:892.128 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:41:892.178 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:41:892.201 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:41:892.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:41:892.241 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:41:892.267 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:41:892.298 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:41:892.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:41:892.325 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:41:969.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:41:969.321 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:41:969.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:41:969.391 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:41:969.614 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:41:969.653 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:41:969.675 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:41:969.694 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:42:050.419 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:42:050.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:42:050.584 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:42:050.621 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:42:050.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:42:050.846 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:42:050.880 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:42:050.894 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:42:050.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:42:050.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:42:233.916 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:42:235.219 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:46:892.241 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:47:323.559 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:47:328.991 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:47:329.046 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:47:329.059 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:47:329.095 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:47:329.124 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:47:329.166 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:47:329.209 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:47:329.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:47:329.262 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:47:329.290 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:47:329.322 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:47:329.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:47:329.379 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:47:329.408 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:47:329.440 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:47:329.476 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:47:329.492 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:47:329.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:47:329.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:47:329.537 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:47:329.562 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:47:329.576 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:47:329.589 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:47:329.601 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:47:329.615 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:47:329.628 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:47:329.640 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:47:329.706 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:52:329.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:52:329.255 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:52:329.302 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:52:329.341 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:52:329.508 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:52:329.561 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:52:329.582 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:52:329.611 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:52:329.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:52:329.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:52:329.729 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:52:329.745 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:52:329.755 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:52:329.779 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:52:329.803 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:52:329.826 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:52:329.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:52:330.196 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:52:330.227 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:52:330.255 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:52:330.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:52:330.312 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:52:330.325 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:52:330.338 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:52:330.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:52:330.363 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:52:330.376 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:52:330.401 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:52:330.421 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:52:330.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:10:55:766.282 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:55:766.382 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SANDESH: Send FAILED: 1737943855766158 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 514 dequeues = 514 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 564 dequeues = 564 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943835321754 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943835322043 ] max_sm_queue_count = 8 ] 2025-01-27 Mon 02:10:55:766.395 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:55:766.405 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:55:766.538 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: SANDESH: Send FAILED: 1737943855766255 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 189, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 96, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:10:55:766.575 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093905950464, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:57:329.372 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:10:57:329.501 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:10:57:329.555 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a55840 2025-01-27 Mon 02:10:57:329.587 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a550d0 2025-01-27 Mon 02:10:57:329.605 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:10:57:329.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a4e070 2025-01-27 Mon 02:10:57:329.758 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:10:57:329.801 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54cd0 2025-01-27 Mon 02:10:57:329.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:10:57:329.846 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a54620 2025-01-27 Mon 02:10:57:330.337 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:10:57:330.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:10:57:330.436 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:10:57:330.463 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:10:57:330.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:10:57:330.521 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:10:57:330.554 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:10:57:330.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:10:57:330.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:10:57:330.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:10:57:330.678 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58fc0 2025-01-27 Mon 02:10:57:330.697 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a59900 2025-01-27 Mon 02:10:57:330.710 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5a240 2025-01-27 Mon 02:10:57:330.823 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5ab80 2025-01-27 Mon 02:10:57:330.839 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57d40 2025-01-27 Mon 02:10:57:330.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a5b4c0 2025-01-27 Mon 02:10:57:330.876 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a58680 2025-01-27 Mon 02:10:57:330.889 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56ac0 2025-01-27 Mon 02:10:57:330.902 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a56180 2025-01-27 Mon 02:10:57:330.929 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: -1 0x2a57400 2025-01-27 Mon 02:11:00:750.365 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:00:750.442 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943860750334 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-01-27 Mon 02:11:00:750.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-01-27 Mon 02:11:02:329.622 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a55840 2025-01-27 Mon 02:11:02:329.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a55840 2025-01-27 Mon 02:11:02:329.806 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:329.835 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:329.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a550d0 2025-01-27 Mon 02:11:02:329.905 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a4e070 2025-01-27 Mon 02:11:02:329.980 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a550d0 2025-01-27 Mon 02:11:02:329.988 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:330.009 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:330.018 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a4e070 2025-01-27 Mon 02:11:02:330.023 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:330.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:330.039 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54cd0 2025-01-27 Mon 02:11:02:330.080 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a54620 2025-01-27 Mon 02:11:02:330.141 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:330.153 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:330.190 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a54cd0 2025-01-27 Mon 02:11:02:330.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:330.208 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:330.217 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a54620 2025-01-27 Mon 02:11:02:330.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:330.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:330.256 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:330.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:330.285 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:330.298 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:330.307 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:333.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:02:333.052 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SANDESH: Send FAILED: 1737943862330358 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:02:333.068 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:02:333.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:333.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:333.980 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:333.991 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:334.004 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:334.013 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:334.021 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58fc0 2025-01-27 Mon 02:11:02:334.081 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a59900 2025-01-27 Mon 02:11:02:334.111 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5a240 2025-01-27 Mon 02:11:02:334.135 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5ab80 2025-01-27 Mon 02:11:02:334.162 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57d40 2025-01-27 Mon 02:11:02:334.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a5b4c0 2025-01-27 Mon 02:11:02:334.213 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a58680 2025-01-27 Mon 02:11:02:334.235 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56ac0 2025-01-27 Mon 02:11:02:334.261 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a56180 2025-01-27 Mon 02:11:02:334.285 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a57400 2025-01-27 Mon 02:11:02:334.349 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:334.371 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:334.404 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a58fc0 2025-01-27 Mon 02:11:02:334.412 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.421 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.436 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a59900 2025-01-27 Mon 02:11:02:334.442 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.450 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.458 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5a240 2025-01-27 Mon 02:11:02:334.463 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.471 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.479 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5ab80 2025-01-27 Mon 02:11:02:334.484 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.491 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.499 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a5b4c0 2025-01-27 Mon 02:11:02:334.504 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.512 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a58680 2025-01-27 Mon 02:11:02:334.585 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.594 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.603 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a57d40 2025-01-27 Mon 02:11:02:334.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.616 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a56ac0 2025-01-27 Mon 02:11:02:334.644 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.825 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a56180 2025-01-27 Mon 02:11:02:334.835 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.860 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: RAC_Connect status: 0 0x2a57400 2025-01-27 Mon 02:11:02:334.866 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:334.874 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:334.907 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:334.917 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:334.948 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:334.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:334.974 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:334.987 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:334.996 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:335.008 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:335.018 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:335.030 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:335.039 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:337.701 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:02:337.737 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943862335070 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:02:337.746 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:02:337.765 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:337.784 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:337.799 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:337.814 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:337.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:337.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:337.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:337.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:337.899 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:337.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:337.920 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:337.934 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:337.947 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:337.956 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:337.990 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:341.132 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:02:341.173 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: SANDESH: Send FAILED: 1737943862338010 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:02:341.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094715455232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:02:341.201 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:341.220 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:341.233 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:341.246 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:05:323.633 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:11:05:323.837 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:11:05:323.932 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:05:323.961 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943865323699 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:05:323.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:05:323.992 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:05:324.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943865323920 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 249, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 102, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:11:05:324.151 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:05:324.602 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:11:05:324.612 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:11:05:326.029 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:11:05:326.040 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:11:05:326.151 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:11:05:326.184 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Idle 2025-01-27 Mon 02:11:05:326.251 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:05:326.391 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943865326107 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 249, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 104, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:11:05:326.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:05:326.416 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:05:326.437 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943865326129 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:05:326.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:05:326.455 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:11:05:326.464 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:05:326.593 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: SANDESH: Send FAILED: 1737943865326236 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 250, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 104, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:11:05:326.606 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094719653632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:25:770.381 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 80 without seqno , total = 0 2025-01-27 Mon 02:11:25:770.448 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:25:770.563 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943885770222 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 525 dequeues = 525 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 578 dequeues = 578 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943865326183 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943865326462 ] max_sm_queue_count = 8 ] 2025-01-27 Mon 02:11:25:770.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:25:770.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:25:770.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943885770360 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 250, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 107, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:11:25:770.867 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:35:328.835 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:11:35:329.031 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:11:35:329.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:35:329.172 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SANDESH: Send FAILED: 1737943895328905 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:35:329.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:35:329.190 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:35:329.333 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SANDESH: Send FAILED: 1737943895329128 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 9 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 62 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 250, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 109, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:11:35:329.353 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:35:333.880 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:47078::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:11:35:333.914 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:47078::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:11:35:333.923 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140095093749888, Pid 1]: 10.0.0.37:47078::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:11:35:334.404 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:11:35:334.416 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:11:35:334.451 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit 2025-01-27 Mon 02:11:35:334.462 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 10 2025-01-27 Mon 02:11:35:334.545 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:11:35:334.554 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:11:35:334.591 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943895334444 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:35:334.602 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:11:35:334.614 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:11:35:334.625 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:11:35:334.639 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:11:35:334.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:11:35:334.778 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: SANDESH: Send FAILED: 1737943895334536 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 10 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 63 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 251, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 111, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-01-27 Mon 02:11:35:334.799 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140093914347264, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:11:35:348.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:11:35:348.712 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:11:35:348.722 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:11:35:348.745 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:11:35:348.965 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 10 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3865666 calls = 411 average_bytes = 9405 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 525 dequeues = 525 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 578 dequeues = 578 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943865326183 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943865326462 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 80 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 314 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 63 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 251, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 111, _iter59->first = sent _iter59->second = 418, ] ] max_sm_queue_count = 8 proxy seq 376 2025-01-27 Mon 02:11:35:348.995 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:11:35:349.021 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 59 2025-01-27 Mon 02:11:35:349.036 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:11:35:349.065 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 80 2025-01-27 Mon 02:11:35:349.075 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:11:35:349.082 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:11:35:349.113 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094937483008, Pid 1]: Established 2025-01-27 Mon 02:12:25:775.322 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094698661632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 81 without seqno , total = 0 2025-01-27 Mon 02:13:25:778.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094694463232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 82 without seqno , total = 0 2025-01-27 Mon 02:14:25:789.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094945879808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 83 without seqno , total = 0 2025-01-27 Mon 02:14:58:484.040 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: Processing scm::EvStop in state Established 2025-01-27 Mon 02:14:58:484.127 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: Idle 2025-01-27 Mon 02:14:58:484.221 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:14:58:484.232 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:14:58:484.372 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: SANDESH: Send FAILED: 1737944098484209 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 10 pid = 1 http_port = 8091 start_time = 1737939050191538 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 5498 calls = 10 average_bytes = 549 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 4088169 calls = 432 average_bytes = 9463 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 18 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 84 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 330 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 64 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 251, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 113, _iter59->first = sent _iter59->second = 442, ] ] ] 2025-01-27 Mon 02:14:58:484.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140094660892416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:15:26:492.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: http-server-port 8091 2025-01-27 Mon 02:15:26:493.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Max-tasks 16 2025-01-27 Mon 02:15:26:493.054 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Max-slice 100 2025-01-27 Mon 02:15:26:493.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Collectors 10.0.0.37:8086 2025-01-27 Mon 02:15:26:493.066 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Collectors 10.0.0.32:8086 2025-01-27 Mon 02:15:26:493.071 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Collectors 10.0.0.22:8086 2025-01-27 Mon 02:15:26:493.095 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: ROLE : Generator 2025-01-27 Mon 02:15:26:493.101 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-01-27 Mon 02:15:26:493.107 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2660-1. 2025-01-27 Mon 02:15:26:493.113 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: NODE TYPE : Database 2025-01-27 Mon 02:15:26:493.118 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: INSTANCE ID : 0 2025-01-27 Mon 02:15:26:493.123 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-01-27 Mon 02:15:26:493.129 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-01-27 Mon 02:15:26:493.699 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: HTTP Introspect Init 2025-01-27 Mon 02:15:26:493.807 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-01-27 Mon 02:15:26:498.549 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Sandesh Http Server Port: 8091 2025-01-27 Mon 02:15:26:498.622 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-01-27 Mon 02:15:26:498.634 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-01-27 Mon 02:15:26:498.752 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Idle 2025-01-27 Mon 02:15:26:498.932 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: No Client: 1737944126498845 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944126498763 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:15:26:499.068 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-01-27 Mon 02:15:26:499.083 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-01-27 Mon 02:15:26:499.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Cassandra Servers: 10.0.0.37:9042 10.0.0.32:9042 10.0.0.22:9042 2025-01-27 Mon 02:15:26:499.456 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvStart in state Idle 2025-01-27 Mon 02:15:26:499.530 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Disconnect 2025-01-27 Mon 02:15:26:499.600 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-01-27 Mon 02:15:26:499.841 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:15:26:499.913 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:15:26:499.940 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: SANDESH: Send FAILED: 1737944126499522 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:15:26:499.949 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:15:26:499.958 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:15:26:500.040 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: SANDESH: Send FAILED: 1737944126499589 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944126498763 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:15:26:500.051 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:15:26:500.060 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:15:26:500.131 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: SANDESH: Send FAILED: 1737944126499645 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944126498763 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:15:26:500.141 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:15:26:500.150 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:15:26:500.164 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: SANDESH: Send FAILED: 1737944126499676 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:15:26:500.175 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:15:26:500.183 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:15:26:500.245 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: SANDESH: Send FAILED: 1737944126499904 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944126498763 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:15:26:500.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:15:26:553.357 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-01-27 Mon 02:15:26:556.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:37474::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-01-27 Mon 02:15:26:558.963 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: 10.0.0.37:37474::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:15:26:558.986 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: 10.0.0.37:37474::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:15:26:559.011 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x137f280 2025-01-27 Mon 02:15:26:559.017 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.030 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.041 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x137f980 2025-01-27 Mon 02:15:26:559.047 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.055 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x1380190 2025-01-27 Mon 02:15:26:559.070 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.078 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.087 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x1380a10 2025-01-27 Mon 02:15:26:559.092 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.100 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.109 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x1381290 2025-01-27 Mon 02:15:26:559.115 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.126 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13bf720 2025-01-27 Mon 02:15:26:559.141 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.149 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.158 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13bcca0 2025-01-27 Mon 02:15:26:559.164 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.171 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13bdda0 2025-01-27 Mon 02:15:26:559.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.197 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.206 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13bd520 2025-01-27 Mon 02:15:26:559.211 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.219 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.227 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13be620 2025-01-27 Mon 02:15:26:559.232 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.240 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.249 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13beea0 2025-01-27 Mon 02:15:26:559.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.262 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.271 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13bffa0 2025-01-27 Mon 02:15:26:559.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.283 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13c0820 2025-01-27 Mon 02:15:26:559.297 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.305 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.708 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13c10a0 2025-01-27 Mon 02:15:26:559.717 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.729 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.739 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: RAC_Connect status: 0 0x13c1920 2025-01-27 Mon 02:15:26:559.745 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:15:26:559.753 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:15:26:559.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:559.978 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:560.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:560.035 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:560.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:560.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:560.066 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:560.078 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:560.086 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:560.098 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:560.107 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:560.135 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:560.153 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:560.168 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:560.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:560.233 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:560.268 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:560.333 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:560.352 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:560.360 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:557.692 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:15:26:560.616 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126554944 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:15:26:560.626 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:15:26:560.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:15:26:560.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:15:26:560.680 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit 2025-01-27 Mon 02:15:26:560.699 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-1.:contrail-query-engine:0:Database count 1 2025-01-27 Mon 02:15:26:560.786 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:560.795 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:15:26:560.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126560011 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:15:26:560.824 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:15:26:560.833 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:560.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:15:26:560.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126560127 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:15:26:560.870 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:15:26:560.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:560.884 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:15:26:560.900 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126560674 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:15:26:560.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:15:26:560.914 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:560.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:15:26:560.940 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:560.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:15:26:561.091 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126560775 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737944126498763 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:15:26:561.113 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:15:26:561.218 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.259 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.272 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.281 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.293 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.302 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.336 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.422 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.435 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.449 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.458 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.471 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.479 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.496 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.505 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.516 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:15:26:561.525 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:15:26:561.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.674 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.688 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.701 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:15:26:561.740 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:561.750 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:15:26:561.773 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126561326 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:15:26:561.781 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:15:26:561.790 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:15:26:561.797 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:15:26:561.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SANDESH: Send FAILED: 1737944126561388 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:15:26:561.826 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:15:26:562.992 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:15:26:563.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:15:26:563.030 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:15:26:563.052 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:15:26:563.165 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737944126498763 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-01-27 Mon 02:15:26:563.403 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:15:26:563.432 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-01-27 Mon 02:15:26:563.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:15:26:563.451 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:15:26:563.459 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:15:26:563.483 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: Established 2025-01-27 Mon 02:16:26:499.545 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183261296384, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-01-27 Mon 02:17:26:500.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182539970304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-01-27 Mon 02:18:26:502.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183278950144, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-01-27 Mon 02:19:26:503.240 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183261296384, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-01-27 Mon 02:20:26:503.820 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-01-27 Mon 02:21:26:504.419 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183287346944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-01-27 Mon 02:22:26:505.012 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-01-27 Mon 02:23:26:505.562 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-01-27 Mon 02:24:26:506.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182544168704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-01-27 Mon 02:25:26:507.099 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182539970304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-01-27 Mon 02:26:26:507.990 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182552565504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-01-27 Mon 02:27:26:508.560 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182544168704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-01-27 Mon 02:28:26:509.044 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-01-27 Mon 02:29:26:513.002 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183283148544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-01-27 Mon 02:30:26:513.594 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183278950144, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-01-27 Mon 02:30:43:110.972 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:41746 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:31:12:735.063 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:53990 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:31:26:514.125 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183291545344, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-01-27 Mon 02:31:33:839.310 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:43738 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:31:47:634.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:34256 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:32:26:514.680 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183261296384, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-01-27 Mon 02:33:06:052.352 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:33114 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:33:26:515.195 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-01-27 Mon 02:33:28:620.941 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.37 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-01-27 Mon 02:33:28:621.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 177e2e9a-dc57-11ef-a68a-00000a000025 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-01-27 Mon 02:33:28:621.182 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 :0:json query is: { end_time: 1737945208553545 , enqueue_time: 1737945208620602 , query_metadata: {"enqueue_time": 1737945208620504} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:28:621.344 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-01-27 Mon 02:33:28:621.360 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-01-27 Mon 02:33:28:621.414 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 177e2e9a-dc57-11ef-a68a-00000a000025 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-01-27 Mon 02:33:28:621.484 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182539970304, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 :0:json query is: { end_time: 1737945208553545 , enqueue_time: 1737945208620602 , query_metadata: {"enqueue_time": 1737945208620504} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:28:733.780 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:733.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:733.827 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:733.836 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:733.844 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:733.860 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:733.868 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:733.875 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:733.882 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:733.890 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:744.256 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:744.291 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:744.300 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:744.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:744.316 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 0 and 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:744.364 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:744.377 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 33 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:744.395 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:744.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 33 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:744.422 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:744.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 :0:json query is: { end_time: 1737945208553545 , enqueue_time: 1737945208620602 , query_metadata: {"enqueue_time": 1737945208620504} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:28:744.868 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:745.088 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:745.319 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:745.520 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:745.667 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:745.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:745.939 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.184 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.305 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.546 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.666 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.791 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:746.912 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.036 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.157 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.289 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.448 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.574 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.701 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:747.941 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.060 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.304 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.422 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.663 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.786 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:748.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:749.025 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:749.143 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:749.678 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:749.722 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:749.734 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:749.745 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:749.763 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:749.773 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:749.782 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:749.792 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:749.801 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:28:749.811 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: 177e2e9a-dc57-11ef-a68a-00000a000025 Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:28:750.048 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.184 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.462 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.585 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.711 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.832 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:750.991 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:751.113 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:751.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:751.401 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QEQueryLog: 177e2e9a-dc57-11ef-a68a-00000a000025 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:28:751.670 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-01-27 Mon 02:33:28:752.528 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-01-27 Mon 02:33:28:753.993 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182548367104, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":44} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-01-27 Mon 02:33:28:756.117 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182544168704, Pid 1]: [SYS_INFO]: QELog: Finished: QID 177e2e9a-dc57-11ef-a68a-00000a000025 Table ObjectVNTable Time(ms) 133 RedisTime(ms) 1 MergeTime(ms) 0 Rows 44 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-01-27 Mon 02:33:28:756.336 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2660-1. table = ObjectVNTable query_stats= [ qid = 177e2e9a-dc57-11ef-a68a-00000a000025 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 111,0, 122,0, 128,0, chunk_select_time = 0,0, 0,4, 0,1, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 133 rows = 44 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-01-27 Mon 02:33:28:758.767 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182544168704, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-01-27 Mon 02:33:31:744.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.37 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-01-27 Mon 02:33:31:744.442 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 195abe5a-dc57-11ef-b638-00000a000025 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-01-27 Mon 02:33:31:744.474 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 :0:json query is: { end_time: 1737945211731672 , enqueue_time: 1737945211743830 , query_metadata: {"enqueue_time": 1737945211743460} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "aaafb063-05ab-4ee1-952d-bbef14a80b9f", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:31:744.619 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-01-27 Mon 02:33:31:744.635 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-01-27 Mon 02:33:31:744.680 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 195abe5a-dc57-11ef-b638-00000a000025 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-01-27 Mon 02:33:31:747.429 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 :0:json query is: { end_time: 1737945211731672 , enqueue_time: 1737945211743830 , query_metadata: {"enqueue_time": 1737945211743460} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "aaafb063-05ab-4ee1-952d-bbef14a80b9f", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:31:783.305 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:783.355 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:783.368 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:783.388 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:783.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:783.403 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:783.411 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:783.418 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:783.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:783.432 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:784.820 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:784.847 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:784.849 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:784.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:784.897 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:784.904 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:784.912 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:784.924 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:784.941 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:784.952 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:784.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:784.969 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:785.180 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id aaafb063-05ab-4ee1-952d-bbef14a80b9f src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:31:785.315 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id aaafb063-05ab-4ee1-952d-bbef14a80b9f src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:31:785.477 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id aaafb063-05ab-4ee1-952d-bbef14a80b9f src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:31:785.599 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id aaafb063-05ab-4ee1-952d-bbef14a80b9f src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:31:784.859 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:785.714 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:785.725 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:785.729 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id aaafb063-05ab-4ee1-952d-bbef14a80b9f src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:31:785.733 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:785.949 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:785.963 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:785.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:31:785.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QELog: 195abe5a-dc57-11ef-b638-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:31:786.023 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: [SYS_INFO]: QEQueryLog: 195abe5a-dc57-11ef-b638-00000a000025 :0:json query is: { end_time: 1737945211731672 , enqueue_time: 1737945211743830 , query_metadata: {"enqueue_time": 1737945211743460} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "aaafb063-05ab-4ee1-952d-bbef14a80b9f", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:31:786.376 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-01-27 Mon 02:33:31:786.500 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-01-27 Mon 02:33:31:787.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-01-27 Mon 02:33:31:787.327 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: Finished: QID 195abe5a-dc57-11ef-b638-00000a000025 Table ObjectVMTable Time(ms) 42 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-01-27 Mon 02:33:31:787.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183283148544, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2660-1. table = ObjectVMTable query_stats= [ qid = 195abe5a-dc57-11ef-b638-00000a000025 where = [[{"name": "ObjectId", "value": "aaafb063-05ab-4ee1-952d-bbef14a80b9f", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 40,0, 38,0, 37,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 42 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-01-27 Mon 02:33:31:787.380 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183257097984, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-01-27 Mon 02:34:26:516.365 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182539970304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-01-27 Mon 02:35:26:517.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-01-27 Mon 02:36:26:517.734 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-01-27 Mon 02:37:26:518.409 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183299942144, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-01-27 Mon 02:38:26:519.011 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183265494784, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-01-27 Mon 02:39:26:519.521 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182535771904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-01-27 Mon 02:40:22:274.749 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:50506 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:40:26:520.391 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183295743744, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-01-27 Mon 02:41:26:520.919 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183291545344, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-01-27 Mon 02:42:25:101.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140183447804032, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.37:8091::10.0.0.37:52686 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:42:26:521.442 UTC cn-jenkins-deploy-platform-ansible-os-2660-1 [Thread 140182556763904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0