2025-02-10 Mon 00:49:26:013.702 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: http-server-port 8091 2025-02-10 Mon 00:49:26:013.796 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Max-tasks 16 2025-02-10 Mon 00:49:26:013.805 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Max-slice 100 2025-02-10 Mon 00:49:26:013.811 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Collectors 10.0.0.17:8086 2025-02-10 Mon 00:49:26:013.816 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Collectors 10.0.0.12:8086 2025-02-10 Mon 00:49:26:013.821 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Collectors 10.0.0.30:8086 2025-02-10 Mon 00:49:26:013.841 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: ROLE : Generator 2025-02-10 Mon 00:49:26:013.847 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-10 Mon 00:49:26:013.852 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2705-3. 2025-02-10 Mon 00:49:26:013.857 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-10 Mon 00:49:26:013.862 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-10 Mon 00:49:26:013.867 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-10 Mon 00:49:26:013.873 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-10 Mon 00:49:26:014.713 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: HTTP Introspect Init 2025-02-10 Mon 00:49:26:014.862 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-10 Mon 00:49:26:015.024 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Sandesh Http Server Port: 8091 2025-02-10 Mon 00:49:26:015.075 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-10 Mon 00:49:26:015.085 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-10 Mon 00:49:26:015.176 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Idle 2025-02-10 Mon 00:49:26:015.342 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: No Client: 1739148566015267 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:49:26:015.473 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-10 Mon 00:49:26:015.486 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-10 Mon 00:49:26:015.498 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Cassandra Servers: 10.0.0.17:9042 10.0.0.12:9042 10.0.0.30:9042 2025-02-10 Mon 00:49:26:016.868 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvStart in state Idle 2025-02-10 Mon 00:49:26:016.935 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Disconnect 2025-02-10 Mon 00:49:26:017.008 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-10 Mon 00:49:26:017.281 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-10 Mon 00:49:26:017.371 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:49:26:017.398 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SANDESH: Send FAILED: 1739148566016927 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-10 Mon 00:49:26:017.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:49:26:017.422 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:49:26:017.540 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SANDESH: Send FAILED: 1739148566016997 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:49:26:017.559 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:49:26:017.569 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:49:26:017.641 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SANDESH: Send FAILED: 1739148566017055 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:49:26:017.654 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:49:26:017.663 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:49:26:017.677 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SANDESH: Send FAILED: 1739148566017075 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-10 Mon 00:49:26:017.684 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:49:26:017.692 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:49:26:017.782 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SANDESH: Send FAILED: 1739148566017358 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:49:26:017.796 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:49:26:020.760 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-10 Mon 00:49:26:020.784 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:26:020.835 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:49:26:020.898 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:49:26:020.926 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: SANDESH: Send FAILED: 1739148566020815 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-10 Mon 00:49:26:020.935 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:49:31:023.324 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-10 Mon 00:49:31:023.365 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:31:023.401 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:49:36:024.692 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-10 Mon 00:49:36:024.816 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:36:024.839 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:49:41:026.510 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-10 Mon 00:49:41:026.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:41:026.591 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:49:46:198.423 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:49:46:198.475 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:46:198.497 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:49:51:239.383 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:49:51:239.430 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:51:239.455 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:49:56:278.638 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:49:56:278.683 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:49:56:278.706 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:50:01:325.559 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:50:01:325.616 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:50:01:325.638 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:50:06:365.458 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:50:06:365.505 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:50:06:365.526 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:50:11:391.266 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:50:11:391.310 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:50:11:391.331 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:50:16:421.816 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-10 Mon 00:50:16:421.869 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-10 Mon 00:50:16:421.892 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:50:21:462.270 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-10 Mon 00:50:21:462.331 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-10 Mon 00:50:26:486.720 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-10 Mon 00:50:31:516.005 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:50:31:516.053 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-10 Mon 00:50:31:516.089 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: -(-1):EvTcpConnectFail 2025-02-10 Mon 00:50:31:516.108 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x14035f0 2025-02-10 Mon 00:50:31:516.125 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.075 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631515912 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-10 Mon 00:50:31:516.140 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.143 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:31:516.158 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1403d10 2025-02-10 Mon 00:50:31:516.176 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.180 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-10 Mon 00:50:31:516.186 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.190 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Connect : EvTcpConnectFail 2025-02-10 Mon 00:50:31:516.212 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1405210 2025-02-10 Mon 00:50:31:516.223 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.233 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.243 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x13fd1d0 2025-02-10 Mon 00:50:31:516.248 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.257 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.267 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x14043b0 2025-02-10 Mon 00:50:31:516.272 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.281 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.290 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1405b50 2025-02-10 Mon 00:50:31:516.295 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.310 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.319 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1406490 2025-02-10 Mon 00:50:31:516.324 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.332 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.341 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x14048e0 2025-02-10 Mon 00:50:31:516.345 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 00:50:31:516.347 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.371 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.380 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1407fe0 2025-02-10 Mon 00:50:31:516.385 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.393 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.398 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Idle 2025-02-10 Mon 00:50:31:516.401 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1406dd0 2025-02-10 Mon 00:50:31:516.417 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.426 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.436 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1407710 2025-02-10 Mon 00:50:31:516.441 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.450 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.459 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1408920 2025-02-10 Mon 00:50:31:516.465 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.470 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:516.473 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.493 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1409260 2025-02-10 Mon 00:50:31:516.501 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.513 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.522 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x1409ba0 2025-02-10 Mon 00:50:31:516.527 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.538 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.547 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: 0 0x140a4e0 2025-02-10 Mon 00:50:31:516.552 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: Connected to REDIS... 2025-02-10 Mon 00:50:31:516.561 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 00:50:31:516.593 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631516293 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 40 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 41, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:50:31:516.615 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:50:31:516.626 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:516.646 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631516322 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-10 Mon 00:50:31:516.655 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:31:516.666 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 00:50:31:516.714 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:516.819 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631516459 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 48, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:50:31:516.834 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:50:31:516.848 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:516.887 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:516.925 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:516.952 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:516.973 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631516861 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739148631516396 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739148631516713 ] max_sm_queue_count = 5 ] 2025-02-10 Mon 00:50:31:516.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:50:31:516.985 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.009 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:516.991 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:517.053 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.076 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.093 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.106 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.122 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.133 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.134 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631516912 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 53 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:50:31:517.188 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:50:31:517.175 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.209 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:517.224 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.248 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148631516934 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 00:50:31:517.256 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:31:517.293 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:517.325 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: SANDESH: Send FAILED: 1739148631517270 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.17:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 00:50:31:517.332 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:31:517.295 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.474 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.490 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.506 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.542 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.568 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:517.608 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SANDESH: Send FAILED: 1739148631517524 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 00:50:31:517.617 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:31:517.712 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.723 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.736 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.745 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.757 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.766 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.778 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.787 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.803 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.812 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.824 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.832 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.844 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.853 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.865 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.873 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.885 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 00:50:31:517.894 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 00:50:31:517.912 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.947 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.962 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.975 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:517.983 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498928883456, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 00:50:31:518.030 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498928883456, Pid 1]: SANDESH: Send FAILED: 1739148631517930 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 00:50:31:518.046 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498928883456, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:31:517.990 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:518.218 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:518.230 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:518.243 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:31:518.254 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 00:50:32:516.541 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 00:50:32:516.750 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Connect : Start Connect timer 10.0.0.17:8086 2025-02-10 Mon 00:50:32:516.859 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:50:32:516.888 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SANDESH: Send FAILED: 1739148632516615 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 00:50:32:516.898 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 00:50:32:516.907 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 00:50:32:517.027 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SANDESH: Send FAILED: 1739148632516848 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:50:32:517.049 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 00:50:32:518.901 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:58912::10.0.0.17:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-10 Mon 00:50:32:518.932 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: 10.0.0.30:58912::10.0.0.17:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 00:50:32:518.940 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: 10.0.0.30:58912::10.0.0.17:8086(-1):EvTcpConnected 2025-02-10 Mon 00:50:32:518.990 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 00:50:32:519.014 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 00:50:32:519.052 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: ClientInit 2025-02-10 Mon 00:50:32:519.085 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 1 2025-02-10 Mon 00:50:32:519.166 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 00:50:32:519.175 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 00:50:32:519.198 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148632519045 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 00:50:32:519.219 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 00:50:32:519.228 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 00:50:32:519.235 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 00:50:32:519.246 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 00:50:32:519.252 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 00:50:32:519.355 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SANDESH: Send FAILED: 1739148632519155 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 00:50:32:519.377 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 00:50:32:522.750 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: OnMessage control in state: ClientInit 2025-02-10 Mon 00:50:32:522.803 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-10 Mon 00:50:32:522.814 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-10 Mon 00:50:32:522.838 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: Received Ctrl Message with size 0 2025-02-10 Mon 00:50:32:523.007 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739148631516396 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739148631516713 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-10 Mon 00:50:32:523.035 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 00:50:32:523.061 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-10 Mon 00:50:32:523.073 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 00:50:32:523.081 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-10 Mon 00:50:32:523.088 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-10 Mon 00:50:32:523.120 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: Established 2025-02-10 Mon 00:50:40:322.499 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: OnMessage control in state: Established 2025-02-10 Mon 00:50:40:322.581 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-10 Mon 00:50:40:322.591 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Established : EvSandeshMessageRecv 2025-02-10 Mon 00:50:40:322.635 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: Received Ctrl Message with size 3 2025-02-10 Mon 00:50:40:322.652 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-10 Mon 00:50:40:322.663 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-10 Mon 00:50:40:322.674 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-10 Mon 00:50:40:322.681 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-10 Mon 00:51:01:517.228 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-10 Mon 00:52:01:517.787 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-10 Mon 00:53:01:518.320 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-10 Mon 00:53:33:228.139 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8091::10.0.0.30:59838 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-10 Mon 00:53:50:723.691 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8091::10.0.0.30:59514 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-10 Mon 00:54:01:519.068 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-10 Mon 00:55:01:520.585 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499055978240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-10 Mon 00:56:01:521.386 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498966660864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-10 Mon 00:57:01:522.392 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498958264064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-10 Mon 00:58:01:524.440 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-10 Mon 00:59:01:526.059 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-10 Mon 01:00:01:527.379 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498983450368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-10 Mon 01:01:01:528.217 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-10 Mon 01:02:01:529.323 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499055978240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-10 Mon 01:03:01:530.303 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-10 Mon 01:04:01:530.859 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498966660864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-10 Mon 01:05:01:533.277 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140497960032000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-10 Mon 01:06:01:534.657 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-10 Mon 01:07:01:535.692 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499055978240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-10 Mon 01:08:01:537.046 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499064375040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-10 Mon 01:09:01:538.808 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-10 Mon 01:10:01:539.682 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-10 Mon 01:11:01:540.718 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498928883456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-10 Mon 01:12:01:541.789 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498958264064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-10 Mon 01:13:01:542.825 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-10 Mon 01:14:01:543.784 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-10 Mon 01:15:01:544.380 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498983450368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-10 Mon 01:16:01:545.452 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-10 Mon 01:17:01:546.042 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-10 Mon 01:18:01:546.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-10 Mon 01:19:01:547.073 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-10 Mon 01:20:01:548.480 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-10 Mon 01:21:01:548.994 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-10 Mon 01:22:01:549.991 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-10 Mon 01:23:01:550.985 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-10 Mon 01:24:01:551.487 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-10 Mon 01:25:01:551.989 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-10 Mon 01:26:01:552.490 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498983450368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-10 Mon 01:27:01:552.953 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-10 Mon 01:28:01:553.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498954065664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-10 Mon 01:29:01:554.089 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498979251968, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-10 Mon 01:30:01:554.645 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-10 Mon 01:31:01:555.136 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498933081856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-10 Mon 01:32:01:555.698 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-10 Mon 01:33:01:556.275 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498966660864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-10 Mon 01:34:01:556.795 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-10 Mon 01:35:01:557.366 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498983450368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-10 Mon 01:36:01:557.925 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-10 Mon 01:37:01:558.448 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498966660864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-10 Mon 01:38:01:558.952 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499060176640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-10 Mon 01:39:01:559.450 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498958264064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-10 Mon 01:40:01:560.031 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499055978240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-10 Mon 01:41:01:560.559 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498966660864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-10 Mon 01:42:01:561.121 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499068573440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-10 Mon 01:43:01:561.723 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498983450368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-10 Mon 01:44:01:562.280 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140497960032000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-10 Mon 01:45:01:562.819 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-10 Mon 01:46:01:563.415 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499055978240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-10 Mon 01:47:01:563.947 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498928883456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-10 Mon 01:48:01:564.447 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-10 Mon 01:49:01:564.986 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-10 Mon 01:50:01:565.586 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498962462464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-02-10 Mon 01:51:01:572.209 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498983450368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-02-10 Mon 01:52:01:572.844 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499055978240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-02-10 Mon 01:53:01:573.877 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498949867264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-02-10 Mon 01:54:01:574.472 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498966660864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-02-10 Mon 01:55:01:575.043 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498928883456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-02-10 Mon 01:55:50:012.755 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x140a4e0 2025-02-10 Mon 01:55:50:012.813 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:012.954 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1409ba0 2025-02-10 Mon 01:55:50:012.971 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:013.001 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1409260 2025-02-10 Mon 01:55:50:013.012 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:013.042 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1408920 2025-02-10 Mon 01:55:50:013.052 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:013.079 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-10 Mon 01:55:50:013.086 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1407fe0 2025-02-10 Mon 01:55:50:013.095 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:013.360 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x140a4e0 2025-02-10 Mon 01:55:50:013.399 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1409ba0 2025-02-10 Mon 01:55:50:013.412 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1409260 2025-02-10 Mon 01:55:50:013.424 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1408920 2025-02-10 Mon 01:55:50:013.436 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1407fe0 2025-02-10 Mon 01:55:50:059.501 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1407710 2025-02-10 Mon 01:55:50:059.565 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:059.607 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1406dd0 2025-02-10 Mon 01:55:50:059.619 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:059.639 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1406490 2025-02-10 Mon 01:55:50:059.649 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:059.798 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1405b50 2025-02-10 Mon 01:55:50:059.811 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:059.836 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-10 Mon 01:55:50:059.843 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Disconnect status: -1 0x1405210 2025-02-10 Mon 01:55:50:059.851 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:55:50:059.993 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1407710 2025-02-10 Mon 01:55:50:060.015 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1406dd0 2025-02-10 Mon 01:55:50:060.028 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1406490 2025-02-10 Mon 01:55:50:060.608 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1405b50 2025-02-10 Mon 01:55:50:060.630 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140499212245120, Pid 1]: RAC_Connect status: -1 0x1405210 2025-02-10 Mon 01:55:50:171.832 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: Processing scm::EvStop in state Established 2025-02-10 Mon 01:55:50:171.913 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: Idle 2025-02-10 Mon 01:55:50:172.014 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:55:50:172.026 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:55:50:172.175 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: SANDESH: Send FAILED: 1739152550172002 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739148566015187 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3168662 calls = 333 average_bytes = 9515 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 66 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 262 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 113, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 334, ] ] ] 2025-02-10 Mon 01:55:50:172.196 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 140498924685056, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:56:07:366.497 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: http-server-port 8091 2025-02-10 Mon 01:56:07:366.612 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Max-tasks 16 2025-02-10 Mon 01:56:07:366.621 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Max-slice 100 2025-02-10 Mon 01:56:07:366.626 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Collectors 10.0.0.17:8086 2025-02-10 Mon 01:56:07:366.631 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Collectors 10.0.0.12:8086 2025-02-10 Mon 01:56:07:366.636 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Collectors 10.0.0.30:8086 2025-02-10 Mon 01:56:07:366.655 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: ROLE : Generator 2025-02-10 Mon 01:56:07:366.660 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-10 Mon 01:56:07:366.666 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2705-3. 2025-02-10 Mon 01:56:07:366.671 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-10 Mon 01:56:07:366.676 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-10 Mon 01:56:07:366.681 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-10 Mon 01:56:07:366.686 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-10 Mon 01:56:07:367.249 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: HTTP Introspect Init 2025-02-10 Mon 01:56:07:367.354 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-10 Mon 01:56:07:367.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Sandesh Http Server Port: 8091 2025-02-10 Mon 01:56:07:367.442 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-10 Mon 01:56:07:367.450 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-10 Mon 01:56:07:367.522 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Idle 2025-02-10 Mon 01:56:07:367.699 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: No Client: 1739152567367624 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 01:56:07:367.825 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-10 Mon 01:56:07:367.838 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-10 Mon 01:56:07:367.850 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Cassandra Servers: 10.0.0.17:9042 10.0.0.12:9042 10.0.0.30:9042 2025-02-10 Mon 01:56:07:399.448 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvStart in state Idle 2025-02-10 Mon 01:56:07:399.530 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Disconnect 2025-02-10 Mon 01:56:07:399.622 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-10 Mon 01:56:07:399.821 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Connect : Start Connect timer 10.0.0.17:8086 2025-02-10 Mon 01:56:07:399.896 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:56:07:399.917 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152567399521 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-10 Mon 01:56:07:399.927 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:56:07:399.935 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:56:07:400.017 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152567399611 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 01:56:07:400.037 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:56:07:400.047 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:56:07:400.116 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152567399669 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 01:56:07:400.129 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:56:07:400.137 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:56:07:400.151 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152567399689 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-10 Mon 01:56:07:400.158 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:56:07:400.165 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:56:07:400.228 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152567399885 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 01:56:07:400.240 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:56:07:529.292 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-10 Mon 01:56:32:538.515 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21bf830 2025-02-10 Mon 01:56:32:538.555 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:538.564 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:56:32:538.572 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:538.624 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21bbda0 2025-02-10 Mon 01:56:32:538.630 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:538.643 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:538.652 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c0400 2025-02-10 Mon 01:56:32:538.660 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:538.668 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:538.677 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c0b50 2025-02-10 Mon 01:56:32:538.682 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:538.692 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:538.707 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c1470 2025-02-10 Mon 01:56:32:538.714 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:538.630 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: SANDESH: Send FAILED: 1739152592538400 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-10 Mon 01:56:32:538.735 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:56:32:538.724 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:538.978 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c1d70 2025-02-10 Mon 01:56:32:538.984 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:538.992 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:539.001 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c26b0 2025-02-10 Mon 01:56:32:539.006 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:539.013 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:539.022 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c2ff0 2025-02-10 Mon 01:56:32:539.027 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:539.034 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:539.044 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c3930 2025-02-10 Mon 01:56:32:539.048 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:539.056 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:539.064 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c4270 2025-02-10 Mon 01:56:32:539.069 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:539.081 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:541.012 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:48832::10.0.0.17:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-10 Mon 01:56:32:541.032 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:48832::10.0.0.17:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 01:56:32:541.040 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:48832::10.0.0.17:8086(-1):EvTcpConnected 2025-02-10 Mon 01:56:32:541.065 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c4bb0 2025-02-10 Mon 01:56:32:541.071 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:541.080 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:541.092 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c5200 2025-02-10 Mon 01:56:32:541.097 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:541.105 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:541.114 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c5920 2025-02-10 Mon 01:56:32:541.119 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:541.127 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:541.137 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c6120 2025-02-10 Mon 01:56:32:541.142 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:541.149 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:541.174 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c69a0 2025-02-10 Mon 01:56:32:541.179 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 01:56:32:541.201 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 01:56:32:541.369 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 01:56:32:541.401 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit 2025-02-10 Mon 01:56:32:541.417 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 1 2025-02-10 Mon 01:56:32:541.496 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:541.512 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:56:32:541.531 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152592541394 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-10 Mon 01:56:32:541.539 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:56:32:541.547 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:541.554 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 01:56:32:541.565 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:541.571 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 01:56:32:541.706 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152592541486 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 17, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 01:56:32:541.721 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 01:56:32:542.111 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 01:56:32:542.606 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.649 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.666 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.676 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.689 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.727 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.740 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.749 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.760 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.785 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.799 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.808 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.820 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.829 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.856 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.866 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.878 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.887 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.898 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:542.922 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:542.999 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:543.010 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:56:32:543.034 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: SANDESH: Send FAILED: 1739152592542960 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:56:32:543.480 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:56:32:543.494 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:543.712 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:543.742 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:56:32:543.763 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SANDESH: Send FAILED: 1739152592543534 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.30:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 01:56:32:543.770 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:56:32:543.788 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:543.824 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:543.841 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:543.856 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:543.895 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:544.363 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:544.373 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:56:32:544.398 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SANDESH: Send FAILED: 1739152592543915 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 01:56:32:544.405 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:56:32:544.782 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:544.838 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:544.855 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:544.870 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:544.911 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:544.922 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:544.969 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:544.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:544.995 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:545.004 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:545.016 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:545.041 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:545.055 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 01:56:32:545.070 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 01:56:32:545.146 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:545.211 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:545.250 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:56:32:545.260 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:56:32:545.280 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SANDESH: Send FAILED: 1739152592545197 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 01:56:32:545.287 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:56:32:545.881 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:545.918 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:545.932 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 01:56:32:546.477 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: OnMessage control in state: ClientInit 2025-02-10 Mon 01:56:32:546.501 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-10 Mon 01:56:32:546.547 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-10 Mon 01:56:32:546.568 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Received Ctrl Message with size 0 2025-02-10 Mon 01:56:32:546.741 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 17, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-10 Mon 01:56:32:546.772 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 01:56:32:546.810 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-10 Mon 01:56:32:546.823 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 01:56:32:546.830 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-10 Mon 01:56:32:546.838 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-10 Mon 01:56:32:546.879 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: Established 2025-02-10 Mon 01:57:07:368.205 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-10 Mon 01:58:07:368.758 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960846898944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-10 Mon 01:58:34:848.115 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c1470 2025-02-10 Mon 01:58:34:848.645 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:848.794 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c0b50 2025-02-10 Mon 01:58:34:848.811 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:848.835 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c0400 2025-02-10 Mon 01:58:34:848.846 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:848.865 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21bbda0 2025-02-10 Mon 01:58:34:848.875 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:848.898 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-10 Mon 01:58:34:848.911 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21bf830 2025-02-10 Mon 01:58:34:848.919 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:849.118 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:58:34:849.175 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:58:34:849.192 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:58:34:849.206 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:58:34:849.243 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:58:34:866.536 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c4270 2025-02-10 Mon 01:58:34:866.582 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:866.620 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c3930 2025-02-10 Mon 01:58:34:866.630 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:866.739 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:34:866.752 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:866.771 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c26b0 2025-02-10 Mon 01:58:34:866.781 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:866.802 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-10 Mon 01:58:34:866.809 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c1d70 2025-02-10 Mon 01:58:34:866.817 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:34:866.971 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:58:34:866.991 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:58:34:867.006 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:34:867.020 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:58:34:867.034 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:58:34:872.955 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.30:48832::10.0.0.17:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 01:58:34:872.995 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:48832::10.0.0.17:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-10 Mon 01:58:34:873.003 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:48832::10.0.0.17:8086(-1):EvTcpClose 2025-02-10 Mon 01:58:34:873.070 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-10 Mon 01:58:34:873.083 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Established : EvTcpClose 2025-02-10 Mon 01:58:34:873.315 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-10 Mon 01:58:34:873.400 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:34:873.554 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714873182 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = cn-jenkins-deploy-platform-ansible-os-2705-1. collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 134271 calls = 17 average_bytes = 7898 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 18, ] ] ] 2025-02-10 Mon 01:58:34:873.572 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:34:873.586 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:34:873.609 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714873211 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.17:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 01:58:34:873.616 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:34:873.625 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-10 Mon 01:58:34:873.658 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:34:873.798 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714873388 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 134271 calls = 17 average_bytes = 7898 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 18, ] ] ] 2025-02-10 Mon 01:58:34:873.815 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:34:877.123 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:38136::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-10 Mon 01:58:34:877.147 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:38136::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 01:58:34:877.185 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:38136::10.0.0.12:8086(-1):EvTcpConnected 2025-02-10 Mon 01:58:34:879.341 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 01:58:34:879.371 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 01:58:34:879.412 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: ClientInit 2025-02-10 Mon 01:58:34:879.434 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 2 2025-02-10 Mon 01:58:34:879.515 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:34:879.524 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:58:34:879.547 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714879405 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.17:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 01:58:34:879.555 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:58:34:879.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:34:879.570 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 01:58:34:879.581 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:34:879.588 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 01:58:34:879.735 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714879505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 134271 calls = 17 average_bytes = 7898 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 77, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 18, ] ] ] 2025-02-10 Mon 01:58:34:879.756 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 01:58:34:918.233 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:38136::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-10 Mon 01:58:34:918.265 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:38136::10.0.0.12:8086(-1):EvTcpClose 2025-02-10 Mon 01:58:34:918.315 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-10 Mon 01:58:34:918.325 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: ClientInit : EvTcpClose 2025-02-10 Mon 01:58:34:918.446 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Return to idle with 5000 2025-02-10 Mon 01:58:34:918.455 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Idle 2025-02-10 Mon 01:58:34:918.511 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:34:918.669 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714918405 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 135844 calls = 18 average_bytes = 7546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 77, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-02-10 Mon 01:58:34:918.701 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:34:918.714 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:34:918.737 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714918433 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.17:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 01:58:34:918.744 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:34:918.753 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:58:34:918.782 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:34:918.925 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152714918502 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 135844 calls = 18 average_bytes = 7546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 77, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-02-10 Mon 01:58:34:918.941 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:35:048.919 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c69a0 2025-02-10 Mon 01:58:35:048.972 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:35:049.021 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c6120 2025-02-10 Mon 01:58:35:049.037 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:35:049.063 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c5920 2025-02-10 Mon 01:58:35:049.073 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:35:049.098 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c5200 2025-02-10 Mon 01:58:35:049.108 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:35:049.134 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-10 Mon 01:58:35:049.148 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Disconnect status: -1 0x21c4bb0 2025-02-10 Mon 01:58:35:049.179 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-10 Mon 01:58:35:049.847 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:35:049.882 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152715049237 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:35:049.893 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:35:052.485 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:58:35:052.524 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:58:35:052.537 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:58:35:052.549 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:58:35:052.564 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:58:37:369.073 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960876287744, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:37:369.192 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960876287744, Pid 1]: SANDESH: Send FAILED: 1739152717368908 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 40 dequeues = 40 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 49 dequeues = 49 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739152714918454 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739152715049842 ] max_sm_queue_count = 5 ] 2025-02-10 Mon 01:58:37:369.208 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960876287744, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:37:369.229 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960876287744, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:37:369.384 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960876287744, Pid 1]: SANDESH: Send FAILED: 1739152717369022 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 135844 calls = 18 average_bytes = 7546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-02-10 Mon 01:58:37:369.399 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960876287744, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:37:539.605 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:37:539.748 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152717539542 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-10 Mon 01:58:37:539.769 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-10 Mon 01:58:39:849.210 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:58:39:849.392 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:58:39:849.435 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:58:39:849.469 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:58:39:849.497 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:58:39:849.658 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:58:39:849.692 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:58:39:849.713 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:58:39:849.725 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:58:39:849.738 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:58:39:867.033 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:58:39:867.161 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:58:39:867.210 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:58:39:867.241 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:58:39:867.270 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:58:39:867.361 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:58:39:867.388 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:58:39:867.402 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:39:867.415 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:58:39:867.427 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:58:39:918.592 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 01:58:39:918.879 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Connect : Start Connect timer 10.0.0.30:8086 2025-02-10 Mon 01:58:39:919.036 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:39:919.069 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: SANDESH: Send FAILED: 1739152719918682 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:39:919.089 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:39:919.099 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:39:919.240 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: SANDESH: Send FAILED: 1739152719919018 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 135844 calls = 18 average_bytes = 7546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-02-10 Mon 01:58:39:919.255 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:39:921.465 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:58514::10.0.0.30:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-10 Mon 01:58:39:921.501 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:58514::10.0.0.30:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 01:58:39:921.512 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:58514::10.0.0.30:8086(-1):EvTcpConnected 2025-02-10 Mon 01:58:39:921.558 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 01:58:39:921.597 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 01:58:39:921.641 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit 2025-02-10 Mon 01:58:39:921.661 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 3 2025-02-10 Mon 01:58:39:921.752 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:39:921.762 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:58:39:921.785 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152719921625 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:39:921.792 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:58:39:921.801 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:39:921.808 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 01:58:39:921.818 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:39:921.824 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 01:58:39:921.955 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152719921741 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 135844 calls = 18 average_bytes = 7546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-02-10 Mon 01:58:39:921.975 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 01:58:40:052.574 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:58:40:052.749 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:58:40:052.808 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:58:40:052.843 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:58:40:052.854 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:58:40:052.902 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:58:40:052.933 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:58:40:052.967 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:58:40:052.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:58:40:052.994 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:58:41:921.941 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: 10.0.0.30:8086 EvConnectTimerExpired in state ClientInit 2025-02-10 Mon 01:58:41:922.024 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-02-10 Mon 01:58:41:922.191 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: ClientInit : EvConnectTimerExpired 2025-02-10 Mon 01:58:41:922.409 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Return to idle with 5000 2025-02-10 Mon 01:58:41:922.421 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Idle 2025-02-10 Mon 01:58:41:922.481 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:41:922.626 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: SANDESH: Send FAILED: 1739152721922284 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137417 calls = 19 average_bytes = 7232 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-10 Mon 01:58:41:922.654 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:41:922.665 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:41:922.687 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: SANDESH: Send FAILED: 1739152721922321 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:41:922.694 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:41:922.715 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:58:41:922.747 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:41:922.875 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: SANDESH: Send FAILED: 1739152721922471 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137417 calls = 19 average_bytes = 7232 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-10 Mon 01:58:41:922.889 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960842700544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:44:849.751 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:58:44:849.903 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:58:44:849.947 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:58:44:849.993 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:58:44:850.025 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:58:44:850.117 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:58:44:850.150 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:58:44:850.163 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:58:44:850.175 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:58:44:850.203 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:58:44:867.415 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:58:44:867.539 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:58:44:867.576 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:58:44:867.606 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:58:44:867.633 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:58:44:867.753 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:58:44:867.780 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:58:44:867.794 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:44:867.806 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:58:44:867.818 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:58:45:052.877 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:58:45:053.039 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:58:45:053.070 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:58:45:053.105 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:58:45:053.133 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:58:45:053.193 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:58:45:053.235 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:58:45:053.254 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:58:45:053.266 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:58:45:053.282 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:58:46:922.593 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 01:58:46:922.883 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Connect : Start Connect timer 10.0.0.17:8086 2025-02-10 Mon 01:58:46:923.015 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:46:923.056 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152726922684 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:46:923.083 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:46:923.097 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:46:923.251 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152726922991 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137417 calls = 19 average_bytes = 7232 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-10 Mon 01:58:46:923.269 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:46:925.072 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:41816::10.0.0.17:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-10 Mon 01:58:46:925.104 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:41816::10.0.0.17:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 01:58:46:925.120 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:41816::10.0.0.17:8086(-1):EvTcpConnected 2025-02-10 Mon 01:58:46:925.202 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 01:58:46:925.240 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 01:58:46:925.290 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: ClientInit 2025-02-10 Mon 01:58:46:925.315 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 4 2025-02-10 Mon 01:58:46:925.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:46:925.420 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 01:58:46:925.443 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: SANDESH: Send FAILED: 1739152726925280 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:46:925.451 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 01:58:46:925.459 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:46:925.466 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 01:58:46:925.477 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 01:58:46:925.484 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 01:58:46:925.623 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: SANDESH: Send FAILED: 1739152726925399 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137417 calls = 19 average_bytes = 7232 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 20, ] ] ] 2025-02-10 Mon 01:58:46:925.646 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960863692544, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 01:58:47:350.537 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.30:41816::10.0.0.17:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-10 Mon 01:58:47:350.585 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.30:41816::10.0.0.17:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 01:58:47:350.610 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:41816::10.0.0.17:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-10 Mon 01:58:47:350.618 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:41816::10.0.0.17:8086(-1):EvTcpClose 2025-02-10 Mon 01:58:47:350.697 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-10 Mon 01:58:47:350.747 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: ClientInit : EvTcpClose 2025-02-10 Mon 01:58:47:350.897 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Return to idle with 5000 2025-02-10 Mon 01:58:47:350.907 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Idle 2025-02-10 Mon 01:58:47:350.969 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:47:351.121 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: SANDESH: Send FAILED: 1739152727350845 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:58:47:351.147 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:47:351.157 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:47:351.191 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: SANDESH: Send FAILED: 1739152727350882 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:47:351.199 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:47:351.208 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:58:47:351.238 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:47:351.374 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: SANDESH: Send FAILED: 1739152727350958 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:58:47:351.391 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:49:850.855 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:58:49:850.967 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:58:49:851.005 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:58:49:851.036 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:58:49:851.065 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:58:49:851.103 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:58:49:851.129 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:58:49:851.144 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:58:49:851.156 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:58:49:851.403 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:58:49:867.818 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:58:49:867.914 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:58:49:867.969 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:58:49:868.001 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:58:49:868.031 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:58:49:868.071 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:58:49:868.096 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:58:49:868.180 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:49:868.250 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:58:49:868.272 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:58:50:053.134 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:58:50:053.315 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:58:50:053.343 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:58:50:053.394 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:58:50:053.435 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:58:50:053.466 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:58:50:053.499 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:58:50:053.533 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:58:50:053.548 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:58:50:053.573 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:58:52:351.212 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 01:58:52:351.455 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-10 Mon 01:58:52:351.576 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:52:351.607 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: SANDESH: Send FAILED: 1739152732351301 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:52:351.623 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:52:351.632 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:58:52:351.721 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-10 Mon 01:58:52:351.750 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1):EvTcpConnectFail 2025-02-10 Mon 01:58:52:351.791 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: SANDESH: Send FAILED: 1739152732351562 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:58:52:351.807 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:52:351.817 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-10 Mon 01:58:52:351.824 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Connect : EvTcpConnectFail 2025-02-10 Mon 01:58:52:351.917 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 01:58:52:351.959 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Idle 2025-02-10 Mon 01:58:52:352.028 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:52:352.170 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: SANDESH: Send FAILED: 1739152732351874 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:58:52:352.184 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:52:353.047 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:52:353.077 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: SANDESH: Send FAILED: 1739152732351894 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:58:52:353.090 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:58:52:353.099 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:58:52:353.121 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:58:52:353.283 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: SANDESH: Send FAILED: 1739152732352013 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:58:52:353.298 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:58:54:851.190 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:58:54:851.339 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:58:54:851.383 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:58:54:851.444 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:58:54:851.508 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:58:54:851.557 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:58:54:851.568 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:58:54:851.600 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:58:54:851.615 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:58:54:851.668 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:58:54:868.138 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:58:54:868.286 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:58:54:868.334 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:58:54:868.374 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:58:54:868.412 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:58:54:868.519 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:58:54:868.564 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:58:54:868.585 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:54:868.604 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:58:54:868.622 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:58:55:053.392 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:58:55:053.571 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:58:55:053.600 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:58:55:053.637 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:58:55:053.689 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:58:55:053.725 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:58:55:053.741 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:58:55:053.784 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:58:55:053.831 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:58:55:053.853 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:58:59:851.597 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:58:59:851.742 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:58:59:851.802 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:58:59:851.846 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:58:59:851.887 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:58:59:851.946 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:58:59:859.374 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:58:59:859.405 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:58:59:859.424 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:58:59:859.447 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:58:59:868.579 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:58:59:869.898 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:58:59:869.939 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:58:59:869.964 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:58:59:869.992 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:58:59:870.173 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:58:59:870.207 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:58:59:870.221 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:58:59:870.234 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:58:59:870.246 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:00:053.640 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:00:083.282 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:00:083.326 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:00:083.362 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:00:083.411 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:00:083.463 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:00:083.505 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:00:083.530 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:00:083.544 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:00:083.556 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:00:352.097 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 01:59:00:352.342 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Connect : Start Connect timer 10.0.0.30:8086 2025-02-10 Mon 01:59:00:352.365 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-10 Mon 01:59:00:352.393 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1):EvTcpConnectFail 2025-02-10 Mon 01:59:00:352.457 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:59:00:352.488 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: SANDESH: Send FAILED: 1739152740352174 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:59:00:352.500 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:59:00:352.514 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-10 Mon 01:59:00:352.524 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Connect : EvTcpConnectFail 2025-02-10 Mon 01:59:00:352.638 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 01:59:00:352.667 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Idle 2025-02-10 Mon 01:59:00:352.757 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:00:352.993 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: SANDESH: Send FAILED: 1739152740352442 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:00:353.023 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:00:353.039 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:00:353.280 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: SANDESH: Send FAILED: 1739152740352585 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:00:353.305 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:00:353.320 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:00:353.351 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: SANDESH: Send FAILED: 1739152740352615 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:59:00:353.361 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:59:00:353.373 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:59:00:353.400 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:00:353.624 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: SANDESH: Send FAILED: 1739152740352745 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:00:353.646 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960855295744, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:04:859.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:04:859.584 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:04:859.619 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:04:859.644 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:04:859.671 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:04:859.836 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:04:859.879 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:04:859.899 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:04:859.918 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:04:859.933 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:04:870.247 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:04:870.366 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:04:870.405 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:04:870.433 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:04:870.461 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:04:870.556 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:04:870.582 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:04:870.595 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:04:870.693 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:04:870.707 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:05:083.383 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:05:083.581 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:05:083.611 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:05:083.648 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:05:083.678 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:05:083.706 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:05:083.804 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:05:083.828 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:05:083.842 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:05:083.855 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:07:369.321 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-10 Mon 01:59:07:369.433 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:07:369.652 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SANDESH: Send FAILED: 1739152747369176 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 69 dequeues = 69 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 92 dequeues = 92 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739152740352666 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739152740353397 ] max_sm_queue_count = 5 ] 2025-02-10 Mon 01:59:07:369.673 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:07:369.687 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:07:369.856 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SANDESH: Send FAILED: 1739152747369303 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:07:369.882 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:09:859.907 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:09:860.035 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:09:860.070 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:09:860.098 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:09:860.125 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:09:860.163 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:09:860.188 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:09:860.235 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:09:860.249 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:09:860.270 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:09:870.611 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:09:870.717 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:09:870.762 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:09:870.792 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:09:870.822 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:09:870.858 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:09:870.879 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:09:870.893 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:09:870.905 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:09:870.958 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:10:083.645 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:10:083.936 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:10:083.966 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:10:084.002 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:10:084.032 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:10:084.061 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:10:084.095 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:10:084.111 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:10:084.123 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:10:084.135 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:14:860.220 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:14:860.370 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:14:860.416 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:14:860.445 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:14:860.473 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:14:860.512 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:14:860.549 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:14:860.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:14:860.576 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:14:860.589 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:14:870.907 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:14:871.003 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:14:871.035 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:14:871.063 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:14:871.090 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:14:871.145 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:14:871.168 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:14:871.182 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:14:871.209 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:14:871.223 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:15:084.004 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:15:084.195 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:15:084.226 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:15:084.260 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:15:084.288 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:15:084.317 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:15:084.349 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:15:084.364 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:15:084.375 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:15:084.387 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:16:352.799 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 01:59:16:353.035 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Connect : Start Connect timer 10.0.0.17:8086 2025-02-10 Mon 01:59:16:353.169 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:59:16:353.200 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152756352887 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:59:16:353.210 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:59:16:353.219 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:59:16:353.218 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-10 Mon 01:59:16:353.265 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1):EvTcpConnectFail 2025-02-10 Mon 01:59:16:353.375 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152756353140 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:16:353.400 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:16:353.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-10 Mon 01:59:16:353.418 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Connect : EvTcpConnectFail 2025-02-10 Mon 01:59:16:353.507 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 01:59:16:353.542 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Idle 2025-02-10 Mon 01:59:16:353.599 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:16:353.753 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152756353468 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:16:353.770 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:16:353.780 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:16:353.804 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152756353487 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:59:16:353.812 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:59:16:353.820 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:59:16:353.838 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:16:353.970 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152756353590 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:16:353.985 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:19:860.596 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:19:860.763 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:19:860.801 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:19:860.831 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:19:860.857 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:19:860.895 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:19:860.922 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:19:860.936 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:19:860.968 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:19:860.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:19:871.184 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:19:871.240 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:19:871.266 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:19:871.291 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:19:871.317 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:19:871.409 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:19:871.427 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:19:871.439 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:19:871.452 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:19:871.465 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:20:084.262 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:20:084.451 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:20:084.479 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:20:084.514 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:20:084.543 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:20:084.583 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:20:084.617 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:20:084.633 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:20:084.645 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:20:084.656 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:24:860.970 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:24:861.126 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:24:861.183 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:24:861.215 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:24:861.241 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:24:861.311 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:24:861.342 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:24:861.356 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:24:861.368 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:24:861.381 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:24:871.479 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:24:871.623 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:24:871.656 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:24:871.687 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:24:871.714 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:24:871.881 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:24:871.916 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:24:871.938 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:24:871.956 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:24:871.974 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:25:084.527 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:25:084.705 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:25:084.728 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:25:084.776 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:25:084.807 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:25:084.836 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:25:084.868 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:25:084.884 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:25:084.896 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:25:084.908 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:29:861.390 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:29:861.539 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:29:861.583 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:29:861.614 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:29:861.643 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:29:861.729 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:29:861.768 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:29:861.785 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:29:861.798 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:29:861.810 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:29:871.916 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:29:872.003 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:29:872.032 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:29:872.065 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:29:872.092 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:29:872.204 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:29:872.237 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:29:872.250 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:29:872.262 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:29:872.274 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:30:084.775 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:30:084.944 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:30:084.972 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:30:085.005 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:30:085.033 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:30:085.061 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:30:085.097 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:30:085.112 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:30:085.124 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:30:085.136 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:34:861.815 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:34:861.945 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:34:861.986 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:34:862.015 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:34:862.045 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:34:862.132 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:34:862.156 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:34:862.171 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:34:862.183 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:34:862.196 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:34:872.278 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:34:872.382 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:34:872.428 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:34:872.458 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:34:872.485 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:34:872.526 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:34:872.550 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:34:872.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:34:872.575 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:34:872.606 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:35:085.019 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:35:085.219 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:35:085.249 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:35:085.279 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:35:085.303 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:35:085.327 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:35:085.427 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:35:085.491 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:35:085.516 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:35:085.536 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:37:369.630 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:37:369.735 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152777369505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 102 dequeues = 102 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739152756353541 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739152756353836 ] max_sm_queue_count = 5 ] 2025-02-10 Mon 01:59:37:369.780 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:37:369.790 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:37:369.932 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SANDESH: Send FAILED: 1739152777369606 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:37:369.947 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:37:540.258 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:37:540.418 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: SANDESH: Send FAILED: 1739152777540202 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-10 Mon 01:59:37:540.450 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960872089344, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-10 Mon 01:59:39:862.204 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:39:862.340 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:39:862.379 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:39:862.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:39:862.438 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:39:862.591 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:39:862.642 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:39:862.661 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:39:862.675 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:39:862.688 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:39:872.587 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:39:872.702 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:39:872.747 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:39:872.787 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:39:872.815 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:39:872.879 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:39:872.901 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:39:872.915 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:39:872.927 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:39:872.940 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:40:085.289 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:40:085.431 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:40:085.482 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:40:085.534 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:40:085.554 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:40:085.581 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:40:085.618 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:40:085.633 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:40:085.643 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:40:085.679 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:44:862.675 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:44:862.814 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:44:862.855 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:44:862.883 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:44:862.916 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:44:862.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:44:863.005 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:44:863.019 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:44:863.032 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:44:863.044 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:44:872.911 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:44:872.959 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:44:872.988 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:44:873.016 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:44:873.042 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:44:873.119 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:44:873.138 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:44:873.172 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:44:873.196 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:44:873.209 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:45:085.499 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:45:085.675 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:45:085.710 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:45:085.745 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:45:085.781 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:45:085.819 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:45:085.835 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:45:085.848 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:45:085.857 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:45:085.893 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:46:353.968 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 01:59:46:354.182 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-10 Mon 01:59:46:354.289 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:59:46:354.318 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: SANDESH: Send FAILED: 1739152786354041 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:59:46:354.326 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:59:46:354.336 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 01:59:46:354.479 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: SANDESH: Send FAILED: 1739152786354276 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:46:354.504 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:46:354.543 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-10 Mon 01:59:46:354.552 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1):EvTcpConnectFail 2025-02-10 Mon 01:59:46:354.802 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-10 Mon 01:59:46:354.811 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Connect : EvTcpConnectFail 2025-02-10 Mon 01:59:46:354.926 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 01:59:46:354.954 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Idle 2025-02-10 Mon 01:59:46:355.014 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:46:355.156 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152786354884 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:46:355.174 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:46:355.184 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:46:355.205 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152786354906 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 01:59:46:355.212 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 01:59:46:355.220 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 01:59:46:355.229 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 01:59:46:355.361 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152786355004 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 01:59:46:355.376 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 01:59:49:863.054 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:49:863.194 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:49:863.233 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:49:863.263 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:49:863.291 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:49:863.331 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:49:863.359 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:49:863.372 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:49:863.385 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:49:863.398 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:49:873.171 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:49:873.279 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:49:873.310 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:49:873.337 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:49:873.362 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:49:873.451 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:49:873.473 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:49:873.485 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:49:873.498 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:49:873.510 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:50:085.743 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:50:085.906 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:50:085.931 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:50:085.967 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:50:085.996 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:50:086.033 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:50:086.048 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:50:086.060 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:50:086.080 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:50:086.114 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:54:863.403 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:54:863.539 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:54:863.582 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:54:863.615 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:54:863.645 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:54:863.688 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:54:863.714 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:54:863.738 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:54:863.751 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:54:863.765 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:54:873.489 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:54:873.568 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:54:873.599 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:54:873.627 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:54:873.655 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:54:873.738 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:54:873.758 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:54:873.780 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:54:873.794 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:54:873.807 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 01:59:55:085.965 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 01:59:55:086.125 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 01:59:55:086.155 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 01:59:55:086.190 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 01:59:55:086.219 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 01:59:55:086.254 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 01:59:55:086.269 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 01:59:55:086.281 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 01:59:55:086.290 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 01:59:55:086.322 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 01:59:59:863.780 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 01:59:59:863.951 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 01:59:59:864.006 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 01:59:59:864.046 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 01:59:59:864.076 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 01:59:59:864.140 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 01:59:59:864.179 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 01:59:59:864.194 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 01:59:59:864.206 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 01:59:59:864.232 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 01:59:59:873.814 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 01:59:59:873.922 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 01:59:59:873.970 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 01:59:59:874.001 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 01:59:59:874.029 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 01:59:59:874.131 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 01:59:59:874.155 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 01:59:59:874.168 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 01:59:59:874.182 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 01:59:59:874.194 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 02:00:00:086.192 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 02:00:00:086.380 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 02:00:00:086.410 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 02:00:00:086.444 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 02:00:00:086.473 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 02:00:00:086.502 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 02:00:00:086.536 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 02:00:00:086.551 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 02:00:00:086.578 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 02:00:00:086.590 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 02:00:04:864.218 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 02:00:04:864.373 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 02:00:04:864.413 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 02:00:04:864.444 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 02:00:04:864.475 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 02:00:04:864.552 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1470 2025-02-10 Mon 02:00:04:864.582 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0b50 2025-02-10 Mon 02:00:04:864.597 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c0400 2025-02-10 Mon 02:00:04:864.610 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bbda0 2025-02-10 Mon 02:00:04:864.623 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21bf830 2025-02-10 Mon 02:00:04:874.186 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 02:00:04:874.289 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 02:00:04:874.327 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 02:00:04:874.358 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 02:00:04:874.388 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 02:00:04:874.425 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4270 2025-02-10 Mon 02:00:04:874.451 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c3930 2025-02-10 Mon 02:00:04:874.466 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c2ff0 2025-02-10 Mon 02:00:04:874.478 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c26b0 2025-02-10 Mon 02:00:04:874.509 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c1d70 2025-02-10 Mon 02:00:05:086.442 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 02:00:05:086.595 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c69a0 2025-02-10 Mon 02:00:05:086.621 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 02:00:05:086.656 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 02:00:05:086.702 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c6120 2025-02-10 Mon 02:00:05:086.717 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5920 2025-02-10 Mon 02:00:05:086.727 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 02:00:05:086.756 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 02:00:05:086.795 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c5200 2025-02-10 Mon 02:00:05:086.809 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: -1 0x21c4bb0 2025-02-10 Mon 02:00:07:369.838 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-10 Mon 02:00:07:369.897 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:07:369.964 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152807369715 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 113 dequeues = 113 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739152786354953 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739152786355228 ] max_sm_queue_count = 5 ] 2025-02-10 Mon 02:00:07:369.976 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:07:369.985 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:07:370.123 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152807369822 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:07:370.146 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:09:864.626 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1470 2025-02-10 Mon 02:00:09:864.747 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0b50 2025-02-10 Mon 02:00:09:864.801 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c0400 2025-02-10 Mon 02:00:09:864.834 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bbda0 2025-02-10 Mon 02:00:09:864.865 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21bf830 2025-02-10 Mon 02:00:09:864.905 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c1470 2025-02-10 Mon 02:00:09:864.912 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:864.937 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:864.964 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c0b50 2025-02-10 Mon 02:00:09:864.973 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:864.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:864.990 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c0400 2025-02-10 Mon 02:00:09:864.995 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:865.015 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:865.024 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21bbda0 2025-02-10 Mon 02:00:09:865.029 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:865.036 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:865.065 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21bf830 2025-02-10 Mon 02:00:09:865.074 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:865.082 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:866.764 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:866.791 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:866.822 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:866.837 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:866.846 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:866.859 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:866.868 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:866.880 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:866.889 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:866.901 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:866.910 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:866.982 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:867.010 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:867.027 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:867.031 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:09:867.042 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:867.103 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152809866942 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.30:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:09:867.118 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:00:09:874.478 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4270 2025-02-10 Mon 02:00:09:874.578 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c3930 2025-02-10 Mon 02:00:09:874.616 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c2ff0 2025-02-10 Mon 02:00:09:874.646 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c26b0 2025-02-10 Mon 02:00:09:874.675 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c1d70 2025-02-10 Mon 02:00:09:874.708 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c4270 2025-02-10 Mon 02:00:09:874.714 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:874.734 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:874.755 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c3930 2025-02-10 Mon 02:00:09:874.760 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:874.769 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:874.804 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c2ff0 2025-02-10 Mon 02:00:09:874.819 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:874.828 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:874.836 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c26b0 2025-02-10 Mon 02:00:09:874.843 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:874.851 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:874.873 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c1d70 2025-02-10 Mon 02:00:09:874.878 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:09:874.889 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:09:875.114 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:875.128 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:875.160 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:875.174 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:875.187 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:875.198 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:875.207 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:875.425 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:09:875.460 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152809875235 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.30:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:09:875.474 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:00:09:875.499 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:875.516 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:875.530 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:875.539 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:875.551 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:09:875.560 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:09:875.576 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:09:875.589 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:10:086.661 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c69a0 2025-02-10 Mon 02:00:10:086.829 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c69a0 2025-02-10 Mon 02:00:10:086.841 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:10:086.865 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:10:086.875 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c6120 2025-02-10 Mon 02:00:10:086.920 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5920 2025-02-10 Mon 02:00:10:086.957 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c5200 2025-02-10 Mon 02:00:10:087.041 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c6120 2025-02-10 Mon 02:00:10:087.051 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:10:087.066 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:10:087.075 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c5920 2025-02-10 Mon 02:00:10:087.080 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:10:087.098 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:10:087.107 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c5200 2025-02-10 Mon 02:00:10:087.112 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:10:087.120 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:10:087.127 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x21c4bb0 2025-02-10 Mon 02:00:10:087.199 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:10:087.212 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:10:087.252 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: RAC_Connect status: 0 0x21c4bb0 2025-02-10 Mon 02:00:10:087.259 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:00:10:087.268 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:00:10:087.292 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:10:087.310 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:10:087.319 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:10:087.608 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:10:087.639 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: SANDESH: Send FAILED: 1739152810087364 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:10:087.648 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960472549120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:00:10:087.666 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:10:087.689 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:10:087.698 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:10:087.711 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:10:087.720 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:10:087.736 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:10:087.749 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:10:087.761 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:00:10:087.770 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:00:10:087.793 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:00:16:355.071 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 02:00:16:355.331 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Connect : Start Connect timer 10.0.0.30:8086 2025-02-10 Mon 02:00:16:355.361 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-10 Mon 02:00:16:355.394 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: -(-1):EvTcpConnectFail 2025-02-10 Mon 02:00:16:355.431 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:00:16:355.461 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152816355156 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:16:355.469 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:00:16:355.490 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-10 Mon 02:00:16:355.497 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Connect : EvTcpConnectFail 2025-02-10 Mon 02:00:16:355.595 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-10 Mon 02:00:16:355.624 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Idle 2025-02-10 Mon 02:00:16:355.684 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:16:355.852 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152816355419 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:16:355.869 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:16:355.879 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:16:356.008 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152816355544 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:16:356.023 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:16:356.033 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:16:356.054 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152816355575 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:16:356.061 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:00:16:356.069 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 02:00:16:356.087 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:16:356.216 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: SANDESH: Send FAILED: 1739152816355671 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:16:356.232 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960880486144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:37:370.203 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:37:370.332 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152837370018 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 126 dequeues = 126 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739152816355623 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739152816356086 ] max_sm_queue_count = 5 ] 2025-02-10 Mon 02:00:37:370.359 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:37:370.371 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:37:370.519 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: SANDESH: Send FAILED: 1739152837370155 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:37:370.533 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960888882944, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:37:540.647 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:00:37:540.755 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: SANDESH: Send FAILED: 1739152837540598 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-10 Mon 02:00:37:540.785 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-10 Mon 02:00:46:355.731 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-10 Mon 02:00:46:355.968 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Connect : Start Connect timer 10.0.0.17:8086 2025-02-10 Mon 02:00:46:356.083 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:00:46:356.111 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: SANDESH: Send FAILED: 1739152846355809 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:46:356.120 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:00:46:356.129 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:00:46:356.279 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: SANDESH: Send FAILED: 1739152846356069 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:46:356.302 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:00:46:359.079 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:47780::10.0.0.17:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-10 Mon 02:00:46:359.118 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:47780::10.0.0.17:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 02:00:46:359.129 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139961032554624, Pid 1]: 10.0.0.30:47780::10.0.0.17:8086(-1):EvTcpConnected 2025-02-10 Mon 02:00:46:361.363 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 02:00:46:361.405 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 02:00:46:361.458 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: ClientInit 2025-02-10 Mon 02:00:46:361.474 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 5 2025-02-10 Mon 02:00:46:361.592 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:00:46:361.607 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 02:00:46:361.641 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: SANDESH: Send FAILED: 1739152846361448 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:00:46:361.651 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 02:00:46:361.664 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:00:46:361.674 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 02:00:46:361.689 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:00:46:361.699 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 02:00:46:361.970 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: SANDESH: Send FAILED: 1739152846361577 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 21, ] ] ] 2025-02-10 Mon 02:00:46:362.000 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 02:00:46:366.691 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: OnMessage control in state: ClientInit 2025-02-10 Mon 02:00:46:366.747 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-10 Mon 02:00:46:366.759 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-10 Mon 02:00:46:366.793 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Received Ctrl Message with size 0 2025-02-10 Mon 02:00:46:367.200 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138990 calls = 20 average_bytes = 6949 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 126 dequeues = 126 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739152816355623 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739152816356086 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 14 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 106979 calls = 13 average_bytes = 8229 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 21, ] ] max_sm_queue_count = 5 proxy seq 54 2025-02-10 Mon 02:00:46:367.241 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 02:00:46:367.275 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 37 2025-02-10 Mon 02:00:46:367.297 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 02:00:46:367.347 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-02-10 Mon 02:00:46:367.361 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 02:00:46:367.372 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-10 Mon 02:00:46:367.414 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: Established 2025-02-10 Mon 02:01:07:370.400 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960851097344, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-10 Mon 02:02:07:371.338 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960476747520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-10 Mon 02:03:07:372.357 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960464152320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-10 Mon 02:04:07:373.042 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960884684544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-10 Mon 02:05:07:373.672 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960867890944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-10 Mon 02:06:07:374.253 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960859494144, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-10 Mon 02:06:56:669.896 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvStop in state Established 2025-02-10 Mon 02:06:56:669.993 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Idle 2025-02-10 Mon 02:06:56:670.107 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-10 Mon 02:06:56:670.118 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-10 Mon 02:06:56:670.268 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: SANDESH: Send FAILED: 1739153216670089 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739152567367532 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 486049 calls = 54 average_bytes = 9000 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 58, ] ] ] 2025-02-10 Mon 02:06:56:670.293 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139960468350720, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:07:23:914.938 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: http-server-port 8091 2025-02-10 Mon 02:07:23:915.028 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Max-tasks 16 2025-02-10 Mon 02:07:23:915.037 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Max-slice 100 2025-02-10 Mon 02:07:23:915.043 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Collectors 10.0.0.17:8086 2025-02-10 Mon 02:07:23:915.048 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Collectors 10.0.0.12:8086 2025-02-10 Mon 02:07:23:915.053 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Collectors 10.0.0.30:8086 2025-02-10 Mon 02:07:23:915.076 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: ROLE : Generator 2025-02-10 Mon 02:07:23:915.082 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-10 Mon 02:07:23:915.088 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2705-3. 2025-02-10 Mon 02:07:23:915.093 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-10 Mon 02:07:23:915.098 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-10 Mon 02:07:23:915.104 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-10 Mon 02:07:23:915.109 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-10 Mon 02:07:23:915.672 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: HTTP Introspect Init 2025-02-10 Mon 02:07:23:915.775 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-10 Mon 02:07:23:915.842 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Sandesh Http Server Port: 8091 2025-02-10 Mon 02:07:23:915.896 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-10 Mon 02:07:23:915.906 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-10 Mon 02:07:23:916.005 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Idle 2025-02-10 Mon 02:07:23:916.190 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: No Client: 1739153243916116 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739153243916020 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 02:07:23:916.320 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-10 Mon 02:07:23:916.334 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-10 Mon 02:07:23:916.346 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Cassandra Servers: 10.0.0.17:9042 10.0.0.12:9042 10.0.0.30:9042 2025-02-10 Mon 02:07:23:921.992 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvStart in state Idle 2025-02-10 Mon 02:07:23:922.066 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Disconnect 2025-02-10 Mon 02:07:23:922.148 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-10 Mon 02:07:23:922.316 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Connect : Start Connect timer 10.0.0.17:8086 2025-02-10 Mon 02:07:23:922.390 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:07:23:922.417 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: SANDESH: Send FAILED: 1739153243922058 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-10 Mon 02:07:23:922.429 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:07:23:922.439 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:07:23:922.568 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: SANDESH: Send FAILED: 1739153243922135 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739153243916020 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 02:07:23:922.588 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:07:23:922.599 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:07:23:922.700 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: SANDESH: Send FAILED: 1739153243922195 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739153243916020 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 02:07:23:922.717 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:07:23:922.732 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:07:23:922.755 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: SANDESH: Send FAILED: 1739153243922214 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-10 Mon 02:07:23:922.772 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:07:23:922.785 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:07:23:922.865 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: SANDESH: Send FAILED: 1739153243922379 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739153243916020 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 02:07:23:922.888 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-10 Mon 02:07:23:954.358 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-10 Mon 02:07:23:961.402 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x111f4a0 2025-02-10 Mon 02:07:23:961.429 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.445 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.456 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x111fcd0 2025-02-10 Mon 02:07:23:961.462 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.470 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.479 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x1120550 2025-02-10 Mon 02:07:23:961.485 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.492 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.501 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115fa70 2025-02-10 Mon 02:07:23:961.506 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.514 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.522 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x1120dd0 2025-02-10 Mon 02:07:23:961.527 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.535 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.543 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115c7c0 2025-02-10 Mon 02:07:23:961.548 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.555 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.17 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.563 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x11602f0 2025-02-10 Mon 02:07:23:961.568 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.575 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.584 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x1160b70 2025-02-10 Mon 02:07:23:961.588 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.603 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.611 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115cff0 2025-02-10 Mon 02:07:23:961.616 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.624 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.632 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115d870 2025-02-10 Mon 02:07:23:961.637 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.646 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.655 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115e0f0 2025-02-10 Mon 02:07:23:961.660 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.667 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.675 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115e970 2025-02-10 Mon 02:07:23:961.680 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.687 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.695 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x115f1f0 2025-02-10 Mon 02:07:23:961.700 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.707 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.715 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x11614c0 2025-02-10 Mon 02:07:23:961.720 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.727 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:961.735 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: RAC_Connect status: 0 0x1161e10 2025-02-10 Mon 02:07:23:961.740 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: Connected to REDIS... 2025-02-10 Mon 02:07:23:961.748 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.30 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-10 Mon 02:07:23:963.739 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:35816::10.0.0.17:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-10 Mon 02:07:23:963.767 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: 10.0.0.30:35816::10.0.0.17:8086(-1) OnSessionEvent TCP Connected 2025-02-10 Mon 02:07:23:963.775 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: 10.0.0.30:35816::10.0.0.17:8086(-1):EvTcpConnected 2025-02-10 Mon 02:07:23:964.034 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.048 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.090 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.168 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.179 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.206 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.221 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.230 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.242 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.251 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.262 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.271 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.30:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.288 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.304 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.319 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.344 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.353 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.370 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.382 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.390 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.402 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.411 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.423 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.431 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.443 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.452 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.17:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.493 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.509 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.523 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.537 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.553 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.562 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.575 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.588 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.600 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.608 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.619 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.628 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.639 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-10 Mon 02:07:23:964.648 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-10 Mon 02:07:23:964.664 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.684 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.698 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.710 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.723 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-10 Mon 02:07:23:964.795 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-10 Mon 02:07:23:964.827 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243961134 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-10 Mon 02:07:23:964.836 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-10 Mon 02:07:23:964.850 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-10 Mon 02:07:23:964.858 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-10 Mon 02:07:23:964.889 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit 2025-02-10 Mon 02:07:23:964.920 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2705-3.:contrail-query-engine:0:Database count 1 2025-02-10 Mon 02:07:23:965.010 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.020 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.041 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243964080 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-10 Mon 02:07:23:965.048 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.057 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.063 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.081 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243964200 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.17:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:07:23:965.087 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.094 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.101 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.116 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243964487 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:07:23:965.126 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.133 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.140 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.185 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243964679 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:07:23:965.193 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.200 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.207 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.223 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243964882 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-10 Mon 02:07:23:965.229 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-10 Mon 02:07:23:965.236 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.242 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-10 Mon 02:07:23:965.271 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-10 Mon 02:07:23:965.277 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-10 Mon 02:07:23:965.389 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SANDESH: Send FAILED: 1739153243965002 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739153243916020 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-10 Mon 02:07:23:965.403 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-10 Mon 02:07:23:966.626 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: OnMessage control in state: ClientInit 2025-02-10 Mon 02:07:23:966.657 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-10 Mon 02:07:23:966.665 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-10 Mon 02:07:23:966.687 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: Received Ctrl Message with size 0 2025-02-10 Mon 02:07:23:966.808 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739153243916020 collector_name = collector_ip = 10.0.0.17:8086 collector_list= [ [ (*_iter40) = 10.0.0.17:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-10 Mon 02:07:23:966.831 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 02:07:23:966.853 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2705-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.17:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.17:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.30:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.17:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.30:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-10 Mon 02:07:23:966.864 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-10 Mon 02:07:23:966.881 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-10 Mon 02:07:23:966.889 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-10 Mon 02:07:23:966.917 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689744074496, Pid 1]: Established 2025-02-10 Mon 02:08:23:916.748 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689633756928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-10 Mon 02:09:23:917.400 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689625360128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-10 Mon 02:10:23:918.042 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689625360128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-10 Mon 02:11:23:918.892 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-10 Mon 02:12:23:920.135 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689637955328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-10 Mon 02:13:23:921.715 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689723090688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-10 Mon 02:14:23:922.359 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689637955328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-10 Mon 02:15:23:923.040 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689778521856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-10 Mon 02:16:23:923.668 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689637955328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-10 Mon 02:17:23:924.272 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689748272896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-10 Mon 02:18:23:924.816 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689782720256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-10 Mon 02:19:23:925.512 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689778521856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-10 Mon 02:20:23:926.134 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689723090688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-10 Mon 02:21:23:927.226 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689752471296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-10 Mon 02:22:23:927.784 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689770125056, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-10 Mon 02:23:23:928.334 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689633756928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-10 Mon 02:24:23:928.917 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689646352128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-10 Mon 02:25:23:929.480 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689774323456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-10 Mon 02:26:23:930.009 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689637955328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-10 Mon 02:27:23:930.576 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689778521856, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-10 Mon 02:28:23:931.124 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689625360128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-10 Mon 02:29:23:932.631 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689774323456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-10 Mon 02:30:23:933.270 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689642153728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-10 Mon 02:31:23:933.819 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689774323456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-10 Mon 02:32:23:934.367 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689625360128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-10 Mon 02:33:23:934.890 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689752471296, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-10 Mon 02:34:23:935.430 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689774323456, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-10 Mon 02:34:58:660.514 UTC cn-jenkins-deploy-platform-ansible-os-2705-3 [Thread 139689930582144, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8091::10.0.0.30:46602 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256