2025-02-06 Thu 00:50:53:453.371 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: http-server-port 8091 2025-02-06 Thu 00:50:53:453.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Max-tasks 16 2025-02-06 Thu 00:50:53:453.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Max-slice 100 2025-02-06 Thu 00:50:53:453.467 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 00:50:53:453.472 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 00:50:53:453.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 00:50:53:453.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 00:50:53:453.501 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 00:50:53:453.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-2. 2025-02-06 Thu 00:50:53:453.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 00:50:53:453.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 00:50:53:453.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 00:50:53:453.527 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 00:50:53:454.025 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 00:50:53:454.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 00:50:53:454.183 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 00:50:53:454.221 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 00:50:53:454.230 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 00:50:53:454.301 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Idle 2025-02-06 Thu 00:50:53:454.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: No Client: 1738803053454375 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:454.566 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 00:50:53:454.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 00:50:53:454.590 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 00:50:53:458.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 00:50:53:459.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Disconnect 2025-02-06 Thu 00:50:53:459.082 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 00:50:53:459.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 00:50:53:459.336 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:459.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SANDESH: Send FAILED: 1738803053459005 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:459.366 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:53:459.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:459.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SANDESH: Send FAILED: 1738803053459072 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:459.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:459.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:459.546 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SANDESH: Send FAILED: 1738803053459130 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:459.558 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:459.567 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:459.582 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SANDESH: Send FAILED: 1738803053459149 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:459.588 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:53:459.596 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:459.658 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SANDESH: Send FAILED: 1738803053459325 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:459.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:461.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:50:53:461.641 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:50:53:461.686 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:50:53:461.712 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:461.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SANDESH: Send FAILED: 1738803053461669 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:461.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:58:464.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:50:58:464.726 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:50:58:464.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:03:466.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:51:03:466.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:03:466.299 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:08:467.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:51:08:467.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:08:467.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:13:549.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:13:549.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:13:549.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:18:572.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:18:572.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:18:572.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:23:672.456 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:23:672.502 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:23:672.524 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:28:716.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:28:716.826 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:28:716.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:33:737.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:33:737.817 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:33:737.838 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:38:762.385 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:38:762.425 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:38:762.447 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:43:786.481 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:43:786.520 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:43:786.541 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:48:817.591 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:48:817.661 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:53:831.714 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 00:51:58:843.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:58:843.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfedd70 2025-02-06 Thu 00:51:58:843.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.641 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfee6b0 2025-02-06 Thu 00:51:58:843.669 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.679 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfeb830 2025-02-06 Thu 00:51:58:843.694 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.702 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.710 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfeeff0 2025-02-06 Thu 00:51:58:843.715 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.732 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfe7da0 2025-02-06 Thu 00:51:58:843.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.745 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfec400 2025-02-06 Thu 00:51:58:843.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.766 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfecb50 2025-02-06 Thu 00:51:58:843.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.787 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.795 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfed470 2025-02-06 Thu 00:51:58:843.800 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.815 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xfef930 2025-02-06 Thu 00:51:58:843.820 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.828 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.840 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xff0270 2025-02-06 Thu 00:51:58:843.845 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.852 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.861 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xff0bb0 2025-02-06 Thu 00:51:58:843.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.873 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xff14f0 2025-02-06 Thu 00:51:58:843.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.893 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.901 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xff1e30 2025-02-06 Thu 00:51:58:843.906 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.913 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.921 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xff2770 2025-02-06 Thu 00:51:58:843.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.933 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: 0 0xff30b0 2025-02-06 Thu 00:51:58:843.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:843.957 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:843.982 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 00:51:58:843.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 00:51:58:843.610 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118843456 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:51:58:844.060 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:844.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 00:51:58:844.081 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 00:51:58:844.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 00:51:58:844.265 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Idle 2025-02-06 Thu 00:51:58:844.331 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:844.359 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.379 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.396 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118844161 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:844.499 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:844.477 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.528 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.512 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:844.551 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.569 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.569 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118844195 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:51:58:844.588 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:844.603 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 00:51:58:844.620 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:844.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:844.649 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:844.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:844.702 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:844.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:844.759 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:844.780 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.805 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.830 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.850 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.879 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.900 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118844322 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 53, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:844.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:844.951 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.963 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:844.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:844.995 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:844.997 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118844610 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 00:51:58:845.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:845.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:845.044 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:845.059 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:845.081 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118844643 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:845.089 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:845.098 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:845.103 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118844729 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:845.126 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:845.141 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:845.134 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118845124 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:845.209 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:845.197 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.240 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.310 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.326 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:845.312 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:845.620 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118845235 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738803118844264 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738803118845140 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 00:51:58:845.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:845.642 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:845.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SANDESH: Send FAILED: 1738803118845299 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 92 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 94, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:845.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:844.417 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 00:51:59:844.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 00:51:59:844.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:59:844.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: SANDESH: Send FAILED: 1738803119844485 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:59:844.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:59:844.756 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:59:844.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: SANDESH: Send FAILED: 1738803119844695 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:59:844.893 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631543060224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:846.848 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:43738::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 00:51:59:846.870 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: 10.0.0.41:43738::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 00:51:59:846.877 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: 10.0.0.41:43738::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 00:51:59:846.943 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 00:51:59:846.975 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 00:51:59:847.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: ClientInit 2025-02-06 Thu 00:51:59:847.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 00:51:59:847.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:847.126 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 00:51:59:847.150 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: SANDESH: Send FAILED: 1738803119847004 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:59:847.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 00:51:59:847.173 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:847.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 00:51:59:847.192 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:847.198 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:847.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: SANDESH: Send FAILED: 1738803119847105 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:59:847.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:851.319 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 00:51:59:851.368 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 00:51:59:851.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 00:51:59:851.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 00:51:59:851.599 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738803118844264 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738803118845140 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-06 Thu 00:51:59:851.621 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 00:51:59:851.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-06 Thu 00:51:59:851.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 00:51:59:851.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:51:59:851.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:51:59:851.700 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: Established 2025-02-06 Thu 00:52:11:455.586 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: OnMessage control in state: Established 2025-02-06 Thu 00:52:11:455.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-06 Thu 00:52:11:455.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: Established : EvSandeshMessageRecv 2025-02-06 Thu 00:52:11:455.715 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: Received Ctrl Message with size 3 2025-02-06 Thu 00:52:11:455.729 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-06 Thu 00:52:11:455.739 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-06 Thu 00:52:11:455.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-06 Thu 00:52:11:455.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:52:28:847.326 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631271761664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 00:53:28:848.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631309539072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 00:54:28:849.546 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631309539072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 00:55:04:324.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.41:59426 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-06 Thu 00:55:20:530.925 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.41:59996 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-06 Thu 00:55:28:850.575 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631267563264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 00:56:28:851.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631280158464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 00:57:28:851.695 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 00:58:28:852.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 00:59:28:854.764 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 01:00:28:856.555 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 01:01:28:857.674 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 01:02:28:860.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-06 Thu 01:03:28:861.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-06 Thu 01:04:28:861.887 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631280158464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-06 Thu 01:05:28:862.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-06 Thu 01:06:28:863.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-06 Thu 01:07:28:863.819 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-06 Thu 01:08:28:864.743 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-06 Thu 01:09:28:865.259 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-06 Thu 01:10:28:866.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-06 Thu 01:11:28:867.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-06 Thu 01:12:28:868.267 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631271761664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-06 Thu 01:13:28:869.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-06 Thu 01:14:28:869.882 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631267563264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-06 Thu 01:15:28:870.880 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631568914176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-06 Thu 01:16:28:871.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-06 Thu 01:17:28:872.466 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-06 Thu 01:18:28:873.583 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631538861824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-06 Thu 01:19:28:874.080 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-06 Thu 01:20:28:874.612 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631551457024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-06 Thu 01:21:28:877.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-06 Thu 01:22:28:879.407 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-06 Thu 01:23:28:879.937 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-06 Thu 01:24:28:882.265 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631267563264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-06 Thu 01:25:28:883.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631271761664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-06 Thu 01:26:28:883.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631309539072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-06 Thu 01:27:28:884.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631322130176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-06 Thu 01:28:28:885.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-06 Thu 01:29:28:886.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-06 Thu 01:30:28:886.548 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-06 Thu 01:31:28:888.037 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631568914176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-06 Thu 01:32:28:888.561 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-06 Thu 01:33:28:889.097 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631313737472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-06 Thu 01:34:28:889.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631538861824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-06 Thu 01:35:28:890.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631275960064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-06 Thu 01:36:28:891.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631267563264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-06 Thu 01:37:28:891.548 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-06 Thu 01:38:28:892.066 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631305340672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-06 Thu 01:39:28:892.896 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-06 Thu 01:40:28:894.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631271761664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-06 Thu 01:41:28:896.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631267563264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-06 Thu 01:42:28:897.693 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631564715776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-06 Thu 01:43:28:898.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631309539072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-06 Thu 01:44:28:899.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631538861824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-06 Thu 01:45:28:899.750 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631322130176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-06 Thu 01:46:28:900.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631322130176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-06 Thu 01:47:28:900.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631547258624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-06 Thu 01:48:28:901.537 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631538861824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-06 Thu 01:49:28:902.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631322130176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-06 Thu 01:50:28:903.590 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631280158464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-06 Thu 01:50:39:697.773 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xff30b0 2025-02-06 Thu 01:50:39:697.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:697.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xff2770 2025-02-06 Thu 01:50:39:697.988 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:698.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xff1e30 2025-02-06 Thu 01:50:39:698.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:698.064 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xff14f0 2025-02-06 Thu 01:50:39:698.076 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:698.099 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:50:39:698.106 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xff0bb0 2025-02-06 Thu 01:50:39:698.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:698.336 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xff30b0 2025-02-06 Thu 01:50:39:698.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xff2770 2025-02-06 Thu 01:50:39:698.397 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xff1e30 2025-02-06 Thu 01:50:39:698.409 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xff14f0 2025-02-06 Thu 01:50:39:698.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xff0bb0 2025-02-06 Thu 01:50:39:757.164 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xff0270 2025-02-06 Thu 01:50:39:757.234 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xfef930 2025-02-06 Thu 01:50:39:757.399 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xfeeff0 2025-02-06 Thu 01:50:39:757.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xfee6b0 2025-02-06 Thu 01:50:39:757.481 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:50:39:757.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Disconnect status: -1 0xfedd70 2025-02-06 Thu 01:50:39:757.527 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xff0270 2025-02-06 Thu 01:50:39:757.769 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xfef930 2025-02-06 Thu 01:50:39:757.782 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xfeeff0 2025-02-06 Thu 01:50:39:757.794 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xfee6b0 2025-02-06 Thu 01:50:39:757.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631712585856, Pid 1]: RAC_Connect status: -1 0xfedd70 2025-02-06 Thu 01:50:39:789.700 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: Processing scm::EvStop in state Established 2025-02-06 Thu 01:50:39:789.772 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: Idle 2025-02-06 Thu 01:50:39:789.890 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:50:39:789.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:50:39:790.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: SANDESH: Send FAILED: 1738806639789877 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053454311 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 2855305 calls = 300 average_bytes = 9517 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 59 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 236 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 113, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 301, ] ] ] 2025-02-06 Thu 01:50:39:790.070 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140631534663424, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:53:236.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: http-server-port 8091 2025-02-06 Thu 01:50:53:236.615 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Max-tasks 16 2025-02-06 Thu 01:50:53:236.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Max-slice 100 2025-02-06 Thu 01:50:53:236.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 01:50:53:236.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 01:50:53:236.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 01:50:53:236.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 01:50:53:236.666 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 01:50:53:236.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-2. 2025-02-06 Thu 01:50:53:236.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 01:50:53:236.687 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 01:50:53:236.692 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 01:50:53:236.698 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 01:50:53:237.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 01:50:53:237.494 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 01:50:53:237.553 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 01:50:53:237.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 01:50:53:237.593 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 01:50:53:237.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Idle 2025-02-06 Thu 01:50:53:237.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: No Client: 1738806653237760 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:53:238.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 01:50:53:238.074 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 01:50:53:238.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 01:50:53:238.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 01:50:53:238.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Disconnect 2025-02-06 Thu 01:50:53:238.787 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 01:50:53:239.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:50:53:239.090 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:53:239.112 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806653238698 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:50:53:239.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:50:53:239.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:53:239.209 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806653238776 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:53:239.226 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:53:239.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:53:239.313 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806653238847 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:53:239.326 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:53:240.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:53:240.490 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806653238868 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:50:53:240.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:50:53:240.515 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:50:53:240.586 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806653239079 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:50:53:240.603 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:50:53:292.957 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 01:51:18:309.106 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:51:18:309.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: SANDESH: Send FAILED: 1738806678309025 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:51:18:309.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:51:18:311.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:37322::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:51:18:311.096 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:37322::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:51:18:311.104 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:37322::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 01:51:18:311.139 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x27295f0 2025-02-06 Thu 01:51:18:311.146 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:18:311.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:18:311.175 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x2729d10 2025-02-06 Thu 01:51:18:311.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:18:311.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:18:311.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x27231d0 2025-02-06 Thu 01:51:18:311.205 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:18:311.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:18:311.229 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272a3b0 2025-02-06 Thu 01:51:18:311.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:18:311.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:18:311.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:51:18:311.294 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272a8e0 2025-02-06 Thu 01:51:18:311.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:18:311.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:18:311.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:51:18:311.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:51:18:311.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:51:18:311.365 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:51:18:311.380 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:51:18:311.393 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:51:18:311.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:51:18:311.419 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:51:18:311.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:51:18:311.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:51:18:311.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:51:18:311.577 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: ClientInit 2025-02-06 Thu 01:51:18:311.594 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 01:51:18:311.687 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:18:311.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:18:311.717 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: SANDESH: Send FAILED: 1738806678311569 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:51:18:311.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:18:311.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:18:311.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:51:18:311.757 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:18:311.764 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:51:18:311.882 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: SANDESH: Send FAILED: 1738806678311677 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:51:18:311.896 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:51:18:312.177 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:18:312.197 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:18:312.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:18:312.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:18:312.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:18:312.244 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:18:312.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:18:312.275 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:18:312.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:18:312.302 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:18:312.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:18:312.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:18:312.388 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:18:312.421 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:18:312.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:18:312.441 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:18:312.462 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:18:312.463 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: SANDESH: Send FAILED: 1738806678312337 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:51:18:312.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:18:312.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:51:18:312.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:51:18:312.513 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: SANDESH: Send FAILED: 1738806678312382 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.41:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:51:18:312.524 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:51:18:313.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546915919616, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 01:51:18:313.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 01:51:18:313.803 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 01:51:18:313.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 01:51:18:313.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-06 Thu 01:51:18:313.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:51:18:314.005 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.41:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 6 2025-02-06 Thu 01:51:18:314.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:51:18:314.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:51:18:314.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:51:18:314.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Established 2025-02-06 Thu 01:51:23:311.303 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:51:23:311.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272b210 2025-02-06 Thu 01:51:23:311.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:311.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:311.480 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:51:23:311.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:51:23:311.584 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:51:23:311.625 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:51:23:311.663 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:51:23:311.695 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:51:23:311.722 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:51:23:311.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:51:23:311.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272bb50 2025-02-06 Thu 01:51:23:311.880 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:311.895 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:311.906 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272c490 2025-02-06 Thu 01:51:23:311.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:311.919 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:311.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272cdd0 2025-02-06 Thu 01:51:23:311.933 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:311.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:311.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272d710 2025-02-06 Thu 01:51:23:311.953 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:311.961 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:311.969 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272e050 2025-02-06 Thu 01:51:23:311.973 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:311.982 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:311.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272e990 2025-02-06 Thu 01:51:23:311.994 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:312.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:312.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272f2d0 2025-02-06 Thu 01:51:23:312.020 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:312.028 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:312.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:51:23:312.099 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.124 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.135 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272fc10 2025-02-06 Thu 01:51:23:312.140 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:312.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:312.230 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.287 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x27304e0 2025-02-06 Thu 01:51:23:312.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:51:23:312.301 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:51:23:312.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.432 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.524 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.538 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.547 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.559 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.580 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.589 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.617 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.729 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.754 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.784 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:23:312.832 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:51:23:312.841 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:51:23:312.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:53:238.383 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 01:52:53:239.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 01:53:29:876.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272d710 2025-02-06 Thu 01:53:29:876.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:876.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272cdd0 2025-02-06 Thu 01:53:29:876.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:877.009 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272c490 2025-02-06 Thu 01:53:29:877.026 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:877.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272bb50 2025-02-06 Thu 01:53:29:877.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:877.112 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:29:877.124 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272b210 2025-02-06 Thu 01:53:29:877.137 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:878.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:29:878.782 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:29:878.795 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:29:878.828 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:29:878.841 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:29:963.391 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272a8e0 2025-02-06 Thu 01:53:29:963.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:963.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272a3b0 2025-02-06 Thu 01:53:29:963.501 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:963.638 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x2729d10 2025-02-06 Thu 01:53:29:963.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:963.673 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x27231d0 2025-02-06 Thu 01:53:29:963.683 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:963.707 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:29:963.714 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x27295f0 2025-02-06 Thu 01:53:29:963.722 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:963.896 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:29:963.920 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:29:963.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:29:963.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:53:29:964.013 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:29:998.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:37322::10.0.0.23:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:29:998.429 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:37322::10.0.0.23:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:29:998.492 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-06 Thu 01:53:29:998.507 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Established : EvTcpClose 2025-02-06 Thu 01:53:29:998.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:53:29:998.857 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:29:999.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806809998596 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = cn-jenkins-deploy-platform-ansible-os-2696-1. collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164378 calls = 22 average_bytes = 7471 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 75, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:29:999.023 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:29:999.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:29:999.056 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806809998627 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:53:29:999.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:29:999.077 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-06 Thu 01:53:29:999.105 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:29:999.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806809998845 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164378 calls = 22 average_bytes = 7471 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 75, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:29:999.270 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:001.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:42056::10.0.0.41:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:30:001.173 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:42056::10.0.0.41:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:30:001.270 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:42056::10.0.0.41:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:30:001.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:30:001.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:30:001.354 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit 2025-02-06 Thu 01:53:30:001.372 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 2 2025-02-06 Thu 01:53:30:001.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:001.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:30:001.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806810001349 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:53:30:001.490 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:30:001.498 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:001.505 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:30:001.514 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:001.520 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:001.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806810001439 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 164378 calls = 22 average_bytes = 7471 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-06 Thu 01:53:30:001.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:081.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:42056::10.0.0.41:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:30:081.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:42056::10.0.0.41:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:30:081.990 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:30:082.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:30:082.126 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Return to idle with 5000 2025-02-06 Thu 01:53:30:082.135 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Idle 2025-02-06 Thu 01:53:30:082.191 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:082.349 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806810082084 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 165951 calls = 23 average_bytes = 7215 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:30:082.371 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:082.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:082.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806810082114 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:53:30:082.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:30:082.419 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:30:082.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:082.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806810082183 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 165951 calls = 23 average_bytes = 7215 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:30:082.597 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:114.347 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x27304e0 2025-02-06 Thu 01:53:30:138.342 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:138.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272fc10 2025-02-06 Thu 01:53:30:138.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:138.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272f2d0 2025-02-06 Thu 01:53:30:138.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:138.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272e990 2025-02-06 Thu 01:53:30:138.574 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:138.599 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:30:138.606 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Disconnect status: -1 0x272e050 2025-02-06 Thu 01:53:30:138.615 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:138.823 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:53:30:138.845 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:53:30:138.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:53:30:138.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:53:30:138.884 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:53:30:138.938 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546928514816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:138.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546928514816, Pid 1]: SANDESH: Send FAILED: 1738806810138662 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:30:138.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546928514816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:34:878.822 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:53:34:878.943 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:34:878.964 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:53:34:878.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:53:34:879.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:53:34:879.052 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:53:34:879.087 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:34:879.102 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:34:879.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:34:879.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:34:963.939 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:53:34:964.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:53:34:964.052 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:53:34:964.080 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:53:34:964.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:53:34:964.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:34:964.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:34:964.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:34:964.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:53:34:964.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:35:082.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:35:082.409 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:53:35:082.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:35:082.554 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: SANDESH: Send FAILED: 1738806815082297 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:35:082.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:35:082.572 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:35:082.716 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: SANDESH: Send FAILED: 1738806815082511 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 165951 calls = 23 average_bytes = 7215 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 19, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:35:082.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:084.442 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:57174::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:35:084.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:57174::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:35:084.472 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:57174::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:35:084.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:35:084.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:35:084.600 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: ClientInit 2025-02-06 Thu 01:53:35:084.621 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 3 2025-02-06 Thu 01:53:35:084.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:35:084.719 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:35:084.741 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: SANDESH: Send FAILED: 1738806815084593 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:35:084.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:35:084.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:35:084.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:35:084.775 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:35:084.782 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:084.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: SANDESH: Send FAILED: 1738806815084698 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 165951 calls = 23 average_bytes = 7215 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-06 Thu 01:53:35:084.950 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:085.490 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:57174::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:35:085.504 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:57174::10.0.0.38:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:35:085.540 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:35:085.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:35:085.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Return to idle with 5000 2025-02-06 Thu 01:53:35:086.005 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Idle 2025-02-06 Thu 01:53:35:086.063 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:35:086.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806815085962 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 167524 calls = 24 average_bytes = 6980 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:35:086.230 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:086.240 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:35:086.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806815085986 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:35:086.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:35:086.277 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:35:086.299 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:35:086.432 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806815086054 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 167524 calls = 24 average_bytes = 6980 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:35:086.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:35:138.873 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:53:35:138.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:53:35:138.969 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:53:35:138.997 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:53:35:139.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:53:35:139.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:53:35:139.076 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:53:35:139.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:53:35:139.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:53:35:139.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:53:39:879.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:53:39:879.171 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:39:879.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:53:39:879.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:53:39:879.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:53:39:879.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:53:39:879.327 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:39:879.343 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:39:879.355 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:39:879.367 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:39:964.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:53:39:964.362 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:53:39:964.396 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:53:39:964.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:53:39:964.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:53:39:964.519 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:39:964.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:39:964.557 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:39:964.589 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:53:39:964.603 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:40:086.111 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:40:086.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:53:40:086.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:40:086.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: SANDESH: Send FAILED: 1738806820086492 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:40:086.791 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:40:086.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:40:086.957 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: SANDESH: Send FAILED: 1738806820086728 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 167524 calls = 24 average_bytes = 6980 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:40:086.975 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546093872896, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:40:088.698 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:46180::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:40:088.719 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:46180::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:40:088.733 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:46180::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:40:088.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:40:088.813 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:40:088.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: ClientInit 2025-02-06 Thu 01:53:40:088.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 4 2025-02-06 Thu 01:53:40:088.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:40:088.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:40:088.977 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806820088840 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:40:088.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:40:088.993 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:40:089.000 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:40:089.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:40:089.018 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:40:089.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: SANDESH: Send FAILED: 1738806820088935 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 167524 calls = 24 average_bytes = 6980 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-06 Thu 01:53:40:089.198 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:40:139.109 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:53:40:139.198 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:53:40:139.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:53:40:139.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:53:40:139.295 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:53:40:139.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:53:40:139.361 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:53:40:139.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:53:40:139.388 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:53:40:139.399 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:53:42:803.555 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.41:46180::10.0.0.23:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-06 Thu 01:53:42:805.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.41:46180::10.0.0.23:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:42:805.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:46180::10.0.0.23:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:42:805.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:46180::10.0.0.23:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:42:805.541 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:42:805.575 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:42:805.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Return to idle with 5000 2025-02-06 Thu 01:53:42:805.713 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Idle 2025-02-06 Thu 01:53:42:805.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:805.937 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806822805659 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:42:805.957 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:42:805.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:805.991 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806822805692 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:42:806.001 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:42:806.010 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:42:806.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:806.167 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806822805762 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:42:806.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:44:879.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:53:44:879.377 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:44:879.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:53:44:879.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:53:44:879.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:44:879.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:44:879.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:53:44:879.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:53:44:879.566 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:44:879.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:44:964.586 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:53:44:964.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:53:44:964.725 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:53:44:964.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:53:44:964.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:53:44:964.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:44:964.925 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:44:964.938 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:44:964.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:53:44:964.979 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:45:139.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:53:45:139.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:53:45:139.559 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:53:45:139.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:53:45:139.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:53:45:139.700 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:53:45:139.730 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:53:45:139.743 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:53:45:139.766 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:53:45:139.778 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:53:47:805.836 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:47:806.063 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:53:47:806.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:47:806.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: SANDESH: Send FAILED: 1738806827805909 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:47:806.205 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:47:806.214 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:47:806.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: SANDESH: Send FAILED: 1738806827806154 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:47:806.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:47:806.838 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:53:47:806.848 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:53:47:807.219 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:53:47:807.229 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:53:47:807.339 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:47:807.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Idle 2025-02-06 Thu 01:53:47:807.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:47:807.575 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: SANDESH: Send FAILED: 1738806827807296 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:47:807.595 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:47:807.605 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:47:807.626 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: SANDESH: Send FAILED: 1738806827807318 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:47:807.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:47:807.642 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:47:807.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:47:807.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: SANDESH: Send FAILED: 1738806827807420 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:47:807.794 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546966288128, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:49:879.448 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:53:49:879.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:49:879.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:53:49:879.663 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:53:49:879.691 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:53:49:879.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:49:879.743 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:49:879.755 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:49:879.764 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:53:49:879.795 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:49:964.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:53:49:965.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:53:49:965.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:53:49:965.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:53:49:965.099 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:53:49:965.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:49:965.224 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:49:965.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:49:965.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:49:965.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:53:50:139.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:53:50:139.844 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:53:50:139.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:53:50:139.903 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:53:50:139.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:53:50:139.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:53:50:140.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:53:50:140.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:53:50:140.044 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:53:50:140.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:53:53:239.837 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 01:53:53:239.903 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:53:239.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806833239716 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 64 dequeues = 64 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806827807369 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806827807649 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:53:53:239.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:53:239.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:53:240.135 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: SANDESH: Send FAILED: 1738806833239821 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:53:240.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547049654016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:54:879.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:53:54:879.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:54:879.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:53:54:879.843 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:53:54:879.872 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:53:54:879.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:54:879.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:54:879.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:54:879.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:53:54:879.973 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:54:965.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:53:54:965.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:53:54:965.353 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:53:54:965.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:53:54:965.404 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:53:54:965.486 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:54:965.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:54:965.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:54:965.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:54:965.577 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:53:55:140.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:53:55:140.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:53:55:140.124 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:53:55:140.148 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:53:55:140.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:53:55:140.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:53:55:140.250 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:53:55:140.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:53:55:140.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:53:55:140.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:53:55:807.833 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:55:808.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:53:55:808.173 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:55:808.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:53:55:808.206 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806835807907 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:55:808.210 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:53:55:808.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:55:808.284 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:55:808.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806835808157 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:55:808.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:55:808.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:53:55:808.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:53:55:808.590 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:55:808.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Idle 2025-02-06 Thu 01:53:55:808.684 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:808.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806835808545 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:55:808.863 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:55:808.873 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:808.896 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806835808569 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:55:808.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:55:808.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:55:808.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:55:809.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806835808673 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:53:55:809.087 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:59:879.855 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:53:59:879.993 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:53:59:880.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:53:59:880.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:53:59:880.078 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:53:59:880.124 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:53:59:880.139 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:53:59:880.152 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:53:59:880.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:53:59:880.192 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:53:59:965.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:53:59:965.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:53:59:965.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:53:59:965.687 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:53:59:965.715 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:53:59:965.800 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:53:59:965.831 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:53:59:965.844 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:53:59:965.861 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:53:59:965.874 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:00:140.290 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:00:140.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:00:140.442 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:00:140.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:00:140.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:00:140.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:00:140.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:00:140.569 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:00:140.582 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:00:140.594 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:04:880.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:04:880.223 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:04:880.248 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:04:880.280 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:04:880.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:04:880.337 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:04:880.371 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:04:880.386 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:04:880.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:04:880.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:04:965.858 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:04:965.928 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:04:965.963 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:04:965.991 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:04:966.017 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:04:966.186 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:04:966.206 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:04:966.220 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:04:966.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:04:966.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:05:140.574 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:05:140.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:05:140.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:05:140.695 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:05:140.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:05:140.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:05:140.780 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:05:140.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:05:140.813 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:05:140.825 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:09:880.294 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:09:880.485 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:09:880.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:09:880.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:09:880.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:09:880.604 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:09:880.637 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:09:880.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:09:880.666 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:09:880.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:09:966.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:09:966.372 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:09:966.430 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:09:966.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:09:966.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:09:966.580 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:09:966.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:09:966.635 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:09:966.655 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:09:966.675 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:10:140.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:10:140.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:10:141.066 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:10:141.129 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:10:141.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:10:141.221 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:10:141.272 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:10:141.298 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:10:141.319 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:10:141.337 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:11:808.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:54:11:808.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:54:11:809.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:11:809.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SANDESH: Send FAILED: 1738806851808822 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:11:809.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:11:809.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:11:809.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SANDESH: Send FAILED: 1738806851809064 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:11:809.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:11:809.353 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:54:11:809.379 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:54:11:809.429 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:54:11:809.442 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:54:11:809.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:54:11:809.572 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Idle 2025-02-06 Thu 01:54:11:809.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:11:809.772 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: SANDESH: Send FAILED: 1738806851809499 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:11:809.791 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:11:809.800 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:11:809.832 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: SANDESH: Send FAILED: 1738806851809521 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:11:809.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:11:809.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:54:11:809.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:11:809.999 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: SANDESH: Send FAILED: 1738806851809621 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:11:810.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:14:880.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:14:880.703 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:14:880.727 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:14:880.760 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:14:880.789 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:14:880.826 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:14:880.860 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:14:880.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:14:880.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:14:880.900 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:14:966.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:14:966.732 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:14:966.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:14:966.844 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:14:966.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:14:966.966 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:14:967.028 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:14:967.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:14:967.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:14:977.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:15:141.184 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:15:141.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:15:141.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:15:141.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:15:141.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:15:141.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:15:141.531 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:15:141.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:15:141.583 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:15:141.595 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:19:880.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:19:880.923 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:19:880.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:19:880.981 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:19:881.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:19:881.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:19:881.063 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:19:881.075 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:19:881.085 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:19:881.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:19:967.049 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:19:967.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:19:967.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:19:967.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:19:967.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:19:967.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:19:967.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:19:967.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:19:977.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:19:977.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:20:141.429 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:20:141.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:20:141.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:20:141.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:20:141.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:20:141.714 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:20:141.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:20:141.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:20:141.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:20:141.812 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:23:240.141 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:23:240.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806863239996 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 102 dequeues = 102 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806851809571 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806851809865 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:54:23:240.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:23:240.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:23:240.432 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806863240096 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:23:240.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:23:310.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:23:311.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SANDESH: Send FAILED: 1738806863310898 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-06 Thu 01:54:23:311.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-06 Thu 01:54:24:880.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:24:881.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:24:881.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:24:881.237 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:24:881.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:24:881.296 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:24:881.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:24:881.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:24:881.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:24:881.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:24:967.367 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:24:967.467 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:24:967.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:24:967.542 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:24:967.626 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:24:967.649 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:24:967.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:24:967.741 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:24:977.594 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:24:977.810 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:25:141.686 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:25:141.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:25:141.850 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:25:141.892 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:25:141.913 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:25:141.938 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:25:141.970 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:25:141.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:25:141.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:25:142.030 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:29:881.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:29:881.379 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:29:881.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:29:881.438 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:29:881.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:29:881.490 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:29:881.499 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:29:881.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:29:881.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:29:881.570 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:29:967.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:29:967.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:29:967.821 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:29:967.851 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:29:967.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:29:968.020 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:29:968.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:29:968.046 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:29:977.834 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:29:978.066 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:30:141.936 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:30:142.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:30:142.076 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:30:142.102 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:30:142.131 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:30:142.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:30:142.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:30:142.203 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:30:142.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:30:142.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:34:881.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:34:881.574 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:34:881.608 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:34:881.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:34:881.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:34:881.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:34:881.707 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:34:881.733 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:34:881.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:34:881.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:34:968.052 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:34:968.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:34:968.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:34:968.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:34:968.312 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:34:968.352 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:34:968.366 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:34:968.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:34:978.103 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:34:978.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:35:142.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:35:142.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:35:142.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:35:142.372 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:35:142.409 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:35:142.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:35:142.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:35:142.501 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:35:142.514 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:35:142.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:39:881.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:39:881.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:39:881.825 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:39:881.858 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:39:881.887 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:39:881.923 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:39:881.937 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:39:881.950 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:39:881.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:39:881.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:39:968.377 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:39:968.494 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:39:968.528 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:39:968.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:39:968.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:39:968.674 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:39:968.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:39:968.730 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:39:978.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:39:978.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:40:142.524 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:40:142.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:40:142.692 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:40:142.722 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:40:142.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:40:142.788 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:40:142.820 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:40:142.833 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:40:142.845 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:40:142.858 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:41:809.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:54:41:809.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:54:41:810.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:41:810.041 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806881809750 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:41:810.059 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:41:810.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:41:810.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:54:41:810.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:54:41:810.229 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806881809990 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:41:810.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:41:810.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:54:41:810.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:54:41:810.386 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:54:41:810.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Idle 2025-02-06 Thu 01:54:41:810.485 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:41:810.648 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806881810339 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:41:810.672 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:41:810.687 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:41:810.711 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806881810360 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:41:810.722 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:41:810.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:54:41:810.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:41:810.894 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SANDESH: Send FAILED: 1738806881810474 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:41:810.910 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:44:883.313 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:44:883.438 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:44:883.485 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:44:883.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:44:883.547 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:44:883.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:44:883.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:44:883.622 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:44:883.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:44:883.645 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:44:968.745 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:44:968.880 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:44:968.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:44:968.947 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:44:973.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:44:973.270 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:44:973.285 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:44:973.298 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:44:978.789 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:44:981.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:45:142.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:45:142.999 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:45:143.037 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:45:143.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:45:143.094 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:45:143.132 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:45:143.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:45:143.171 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:45:143.193 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:45:143.206 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:49:883.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:49:883.789 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:49:883.819 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:49:883.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:49:883.887 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:49:883.916 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:49:883.950 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:49:883.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:49:883.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:49:883.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:49:973.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:49:973.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:49:973.396 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:49:973.421 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:49:973.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:49:973.586 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:49:973.598 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:49:973.610 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:49:981.252 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:49:981.448 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:50:143.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:50:143.272 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:50:143.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:50:143.330 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:50:143.365 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:50:143.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:50:143.422 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:50:143.437 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:50:143.449 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:50:143.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:53:240.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 01:54:53:240.390 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:53:240.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SANDESH: Send FAILED: 1738806893240204 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 113 dequeues = 113 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806881810422 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806881810748 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:54:53:240.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:53:240.480 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:53:240.617 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SANDESH: Send FAILED: 1738806893240316 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:54:53:240.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:54:883.864 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:54:884.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:54:884.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:54:884.510 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:54:884.540 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:54:884.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:54:884.601 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:54:884.616 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:54:884.657 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:54:884.669 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:54:973.607 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:54:973.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:54:973.717 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:54:973.744 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:54:973.841 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:54:973.864 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:54:973.878 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:54:973.934 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:54:981.472 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:54:981.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:54:55:143.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:54:55:143.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:54:55:143.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:54:55:143.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:54:55:143.601 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:54:55:143.635 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:54:55:143.655 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:54:55:143.668 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:54:55:143.680 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:54:55:143.693 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:54:59:884.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:54:59:884.675 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:54:59:884.703 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:54:59:884.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:54:59:884.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:54:59:884.791 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:54:59:884.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:54:59:884.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:54:59:884.869 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:54:59:884.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:54:59:973.903 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:54:59:974.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:54:59:974.063 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:54:59:974.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:54:59:974.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:54:59:974.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:54:59:974.339 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:54:59:974.352 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:54:59:981.967 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:54:59:982.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:55:00:143.698 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:55:00:143.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:55:00:143.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:55:00:143.874 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:55:00:143.901 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:55:00:143.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:55:00:143.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:55:00:143.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:55:00:143.991 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:55:00:144.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:55:04:884.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:55:04:884.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272d710 2025-02-06 Thu 01:55:04:884.929 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:55:04:884.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:55:04:884.995 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:55:04:885.029 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272cdd0 2025-02-06 Thu 01:55:04:885.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272c490 2025-02-06 Thu 01:55:04:885.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272bb50 2025-02-06 Thu 01:55:04:885.075 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:55:04:885.107 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272b210 2025-02-06 Thu 01:55:04:974.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:55:04:974.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:55:04:974.493 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:55:04:974.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:55:04:974.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a8e0 2025-02-06 Thu 01:55:04:974.663 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272a3b0 2025-02-06 Thu 01:55:04:974.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x2729d10 2025-02-06 Thu 01:55:04:974.692 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27231d0 2025-02-06 Thu 01:55:04:982.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:55:04:982.480 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27295f0 2025-02-06 Thu 01:55:05:144.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:55:05:144.102 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:55:05:144.143 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:55:05:144.172 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:55:05:144.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:55:05:144.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x27304e0 2025-02-06 Thu 01:55:05:144.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272fc10 2025-02-06 Thu 01:55:05:144.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272f2d0 2025-02-06 Thu 01:55:05:144.295 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e990 2025-02-06 Thu 01:55:05:144.307 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: -1 0x272e050 2025-02-06 Thu 01:55:09:884.964 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272d710 2025-02-06 Thu 01:55:09:885.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272d710 2025-02-06 Thu 01:55:09:885.184 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:885.209 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:885.220 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272cdd0 2025-02-06 Thu 01:55:09:885.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272c490 2025-02-06 Thu 01:55:09:885.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272bb50 2025-02-06 Thu 01:55:09:885.367 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272b210 2025-02-06 Thu 01:55:09:885.436 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272cdd0 2025-02-06 Thu 01:55:09:885.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:885.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:885.463 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272c490 2025-02-06 Thu 01:55:09:885.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:885.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:885.484 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272bb50 2025-02-06 Thu 01:55:09:885.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:885.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:885.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272b210 2025-02-06 Thu 01:55:09:885.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:885.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:885.547 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:885.558 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:885.622 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:885.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:885.651 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:886.183 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:09:886.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: SANDESH: Send FAILED: 1738806909885682 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:09:886.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546924316416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:09:886.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:886.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:886.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:886.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:886.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:886.303 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:886.312 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:886.330 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:886.346 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:886.362 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:974.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a8e0 2025-02-06 Thu 01:55:09:974.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272a3b0 2025-02-06 Thu 01:55:09:974.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2729d10 2025-02-06 Thu 01:55:09:974.832 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27231d0 2025-02-06 Thu 01:55:09:974.931 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272a8e0 2025-02-06 Thu 01:55:09:974.958 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:974.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:975.001 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272a3b0 2025-02-06 Thu 01:55:09:975.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:975.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:975.025 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x2729d10 2025-02-06 Thu 01:55:09:975.030 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:975.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:975.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x27231d0 2025-02-06 Thu 01:55:09:975.077 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:975.085 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:975.219 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:975.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:975.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:975.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:975.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:975.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:975.307 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:975.319 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:975.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:975.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:975.421 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547045455616, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:09:975.462 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547045455616, Pid 1]: SANDESH: Send FAILED: 1738806909975355 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:09:975.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547045455616, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:09:975.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:975.562 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:982.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27295f0 2025-02-06 Thu 01:55:09:982.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x27295f0 2025-02-06 Thu 01:55:09:982.745 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:982.756 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:983.326 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:983.337 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:983.355 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:144.290 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27304e0 2025-02-06 Thu 01:55:10:144.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272fc10 2025-02-06 Thu 01:55:10:144.401 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272f2d0 2025-02-06 Thu 01:55:10:144.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e990 2025-02-06 Thu 01:55:10:144.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x272e050 2025-02-06 Thu 01:55:10:144.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x27304e0 2025-02-06 Thu 01:55:10:144.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:144.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:144.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272fc10 2025-02-06 Thu 01:55:10:144.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:144.530 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:144.538 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272f2d0 2025-02-06 Thu 01:55:10:144.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:144.551 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:144.582 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272e990 2025-02-06 Thu 01:55:10:144.589 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:144.597 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:144.613 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: RAC_Connect status: 0 0x272e050 2025-02-06 Thu 01:55:10:144.618 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:144.626 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:144.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:144.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:144.717 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:144.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:144.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:144.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:144.822 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:144.836 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:144.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:144.858 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:144.870 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:144.879 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:144.895 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:144.908 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:144.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:144.822 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:10:145.141 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806910144771 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:10:145.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:11:810.532 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:55:11:810.823 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:55:11:810.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:11:810.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:55:11:810.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:55:11:810.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806911810688 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:11:811.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:11:811.017 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:11:811.164 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806911810920 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:55:11:811.185 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:11:811.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:55:11:811.203 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:55:11:811.294 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:55:11:811.321 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Idle 2025-02-06 Thu 01:55:11:811.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:11:811.519 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806911811255 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:55:11:811.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:11:811.546 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:11:811.566 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806911811273 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:11:811.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:11:811.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:55:11:811.598 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:11:811.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SANDESH: Send FAILED: 1738806911811369 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:55:11:811.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:23:240.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:23:240.726 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SANDESH: Send FAILED: 1738806923240473 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 126 dequeues = 126 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806911811320 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806911811596 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:55:23:240.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:23:240.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:23:240.901 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SANDESH: Send FAILED: 1738806923240585 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:55:23:240.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:23:311.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546928514816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:23:311.383 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546928514816, Pid 1]: SANDESH: Send FAILED: 1738806923311295 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-06 Thu 01:55:23:311.395 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546928514816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-06 Thu 01:55:41:811.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:55:41:811.718 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:55:41:811.841 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:41:811.872 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806941811551 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:41:811.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:41:811.890 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:41:812.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806941811820 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:55:41:812.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:41:814.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:43646::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:55:41:814.125 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:43646::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:55:41:814.134 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547193325696, Pid 1]: 10.0.0.41:43646::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 01:55:41:814.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:55:41:814.206 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:55:41:814.240 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit 2025-02-06 Thu 01:55:41:814.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 5 2025-02-06 Thu 01:55:41:814.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:41:814.344 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:55:41:814.376 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806941814233 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:41:814.384 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:55:41:814.392 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:41:814.399 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:55:41:814.409 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:41:814.415 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:55:41:814.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: SANDESH: Send FAILED: 1738806941814324 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 26, ] ] ] 2025-02-06 Thu 01:55:41:814.562 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546081277696, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:55:41:836.420 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546907522816, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 01:55:41:836.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 01:55:41:836.492 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 01:55:41:836.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 01:55:41:836.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 169097 calls = 25 average_bytes = 6763 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 126 dequeues = 126 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806911811320 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806911811596 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 18 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 134354 calls = 17 average_bytes = 7903 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 26, ] ] max_sm_queue_count = 5 proxy seq 54 2025-02-06 Thu 01:55:41:836.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:41:836.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 37 2025-02-06 Thu 01:55:41:836.821 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:41:836.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-02-06 Thu 01:55:41:836.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:41:836.874 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:55:41:836.900 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: Established 2025-02-06 Thu 01:55:53:240.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140547041257216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 01:56:53:241.608 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546915919616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 01:57:53:242.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546957891328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 01:58:53:242.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546085476096, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 01:59:53:243.302 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546911721216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 02:00:22:081.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvStop in state Established 2025-02-06 Thu 02:00:22:081.572 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Idle 2025-02-06 Thu 02:00:22:081.674 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 02:00:22:081.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 02:00:22:081.848 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: SANDESH: Send FAILED: 1738807222081662 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738806653237681 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 437416 calls = 51 average_bytes = 8576 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 32 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 55, ] ] ] 2025-02-06 Thu 02:00:22:081.867 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140546920118016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:00:49:432.109 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: http-server-port 8091 2025-02-06 Thu 02:00:49:432.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Max-tasks 16 2025-02-06 Thu 02:00:49:432.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Max-slice 100 2025-02-06 Thu 02:00:49:432.755 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 02:00:49:432.760 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 02:00:49:432.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 02:00:49:432.787 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 02:00:49:432.794 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 02:00:49:432.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-2. 2025-02-06 Thu 02:00:49:432.804 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 02:00:49:432.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 02:00:49:432.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 02:00:49:432.820 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 02:00:49:433.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 02:00:49:433.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 02:00:49:433.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 02:00:49:433.592 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 02:00:49:433.600 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 02:00:49:433.679 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Idle 2025-02-06 Thu 02:00:49:433.841 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: No Client: 1738807249433758 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807249433688 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:00:49:433.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 02:00:49:433.986 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 02:00:49:433.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 02:00:49:441.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 02:00:49:441.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Disconnect 2025-02-06 Thu 02:00:49:441.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 02:00:49:441.729 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 02:00:49:441.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:00:49:441.831 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: SANDESH: Send FAILED: 1738807249441497 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:00:49:441.840 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 02:00:49:441.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:00:49:441.930 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: SANDESH: Send FAILED: 1738807249441563 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807249433688 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:00:49:441.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:00:49:441.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:00:49:442.023 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: SANDESH: Send FAILED: 1738807249441619 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807249433688 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:00:49:442.036 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:00:49:442.044 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:00:49:442.059 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: SANDESH: Send FAILED: 1738807249441637 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:00:49:442.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 02:00:49:442.073 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:00:49:442.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: SANDESH: Send FAILED: 1738807249441791 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807249433688 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:00:49:442.147 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 02:00:49:457.604 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 02:00:49:460.299 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x144d4a0 2025-02-06 Thu 02:00:49:460.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.331 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.342 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148aff0 2025-02-06 Thu 02:00:49:460.348 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x144dcd0 2025-02-06 Thu 02:00:49:460.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.385 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x144e550 2025-02-06 Thu 02:00:49:460.390 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.406 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x144edd0 2025-02-06 Thu 02:00:49:460.411 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.425 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148a7c0 2025-02-06 Thu 02:00:49:460.438 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.454 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148b870 2025-02-06 Thu 02:00:49:460.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.466 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.474 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148c0f0 2025-02-06 Thu 02:00:49:460.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.487 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148c970 2025-02-06 Thu 02:00:49:460.500 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148d1f0 2025-02-06 Thu 02:00:49:460.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.528 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148da70 2025-02-06 Thu 02:00:49:460.541 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.549 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148e2f0 2025-02-06 Thu 02:00:49:460.561 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.569 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.577 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148eb70 2025-02-06 Thu 02:00:49:460.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.593 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.601 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148f4c0 2025-02-06 Thu 02:00:49:460.606 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.613 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:460.621 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: RAC_Connect status: 0 0x148fe10 2025-02-06 Thu 02:00:49:460.626 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: Connected to REDIS... 2025-02-06 Thu 02:00:49:460.634 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 02:00:49:462.583 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:32834::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-06 Thu 02:00:49:462.602 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: 10.0.0.41:32834::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 02:00:49:462.610 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: 10.0.0.41:32834::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 02:00:49:463.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.097 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.111 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.133 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.154 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.162 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.174 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.189 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.212 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.231 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.254 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.332 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.354 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.384 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.500 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.574 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.584 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.596 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.604 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.616 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.625 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.645 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 02:00:49:463.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 02:00:49:463.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.714 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.767 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.780 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 02:00:49:463.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 02:00:49:463.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249460233 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 02:00:49:463.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 02:00:49:463.924 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 02:00:49:463.932 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 02:00:49:463.953 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit 2025-02-06 Thu 02:00:49:463.969 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-2.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 02:00:49:464.049 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249463294 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 02:00:49:464.087 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.102 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249463326 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.41:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:00:49:464.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.138 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.145 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249463349 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:00:49:464.167 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.174 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249463729 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:00:49:464.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.210 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.219 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249463947 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 02:00:49:464.241 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 02:00:49:464.248 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 02:00:49:464.274 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 02:00:49:464.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 02:00:49:464.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: SANDESH: Send FAILED: 1738807249464041 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738807249433688 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 02:00:49:464.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346490689280, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 02:00:49:466.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346365298432, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 02:00:49:469.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 02:00:49:469.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 02:00:49:469.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 02:00:49:469.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738807249433688 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-06 Thu 02:00:49:469.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 02:00:49:469.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-06 Thu 02:00:49:469.739 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 02:00:49:469.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 02:00:49:469.754 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 02:00:49:469.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: Established 2025-02-06 Thu 02:01:49:434.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 02:02:49:435.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346377893632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 02:03:49:435.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 02:04:49:436.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346478094080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 02:05:49:436.769 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346512541440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 02:06:49:437.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346356901632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 02:07:49:438.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346508343040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 02:08:49:439.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346373695232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 02:09:49:440.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 02:10:49:441.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346516739840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 02:11:49:441.835 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-06 Thu 02:12:49:442.537 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346486490880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-06 Thu 02:13:49:443.303 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-06 Thu 02:14:49:443.862 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346356901632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-06 Thu 02:15:18:186.307 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.41:48160 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:15:32:382.880 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.41:57868 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:15:49:444.383 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346373695232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-06 Thu 02:16:49:445.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-06 Thu 02:16:54:500.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.23:34242 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:17:11:723.613 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.23 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-06 Thu 02:17:11:724.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 7958fa68-e430-11ef-895e-00000a000017 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-06 Thu 02:17:11:724.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 :0:json query is: { end_time: 1738808231664840 , enqueue_time: 1738808231723119 , query_metadata: {"enqueue_time": 1738808231723012} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:11:724.447 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-06 Thu 02:17:11:724.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-06 Thu 02:17:11:724.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 7958fa68-e430-11ef-895e-00000a000017 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-06 Thu 02:17:11:725.961 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346373695232, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 :0:json query is: { end_time: 1738808231664840 , enqueue_time: 1738808231723119 , query_metadata: {"enqueue_time": 1738808231723012} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:11:756.365 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:756.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:756.412 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:756.419 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:756.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:756.436 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:756.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:756.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:756.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:756.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:757.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:11:757.921 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:11:758.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:758.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:758.732 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:758.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:758.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:758.755 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:758.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:758.769 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:758.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 UNION between tables of sizes 0 and 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:11:758.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7958fa68-e430-11ef-895e-00000a000017 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:11:758.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 :0:json query is: { end_time: 1738808231664840 , enqueue_time: 1738808231723119 , query_metadata: {"enqueue_time": 1738808231723012} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:11:759.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:11:759.541 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:11:759.686 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QEQueryLog: 7958fa68-e430-11ef-895e-00000a000017 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:11:759.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-06 Thu 02:17:11:760.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-06 Thu 02:17:11:760.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-06 Thu 02:17:11:761.577 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346486490880, Pid 1]: [SYS_INFO]: QELog: Finished: QID 7958fa68-e430-11ef-895e-00000a000017 Table ObjectVMTable Time(ms) 37 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-06 Thu 02:17:11:761.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346486490880, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-06 Thu 02:17:11:761.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346486490880, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2696-2. table = ObjectVMTable query_stats= [ qid = 7958fa68-e430-11ef-895e-00000a000017 where = [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 31,0, 32,0, chunk_select_time = 0,1, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 37 rows = 5 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-06 Thu 02:17:14:796.394 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.23 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-06 Thu 02:17:14:796.918 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 7b2dcd92-e430-11ef-aed0-00000a000017 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-06 Thu 02:17:14:796.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QEQueryLog: 7b2dcd92-e430-11ef-aed0-00000a000017 :0:json query is: { end_time: 1738808234780632 , enqueue_time: 1738808234795848 , query_metadata: {"enqueue_time": 1738808234795735} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:14:797.144 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-06 Thu 02:17:14:797.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-06 Thu 02:17:14:797.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 7b2dcd92-e430-11ef-aed0-00000a000017 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-06 Thu 02:17:14:797.417 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: [SYS_INFO]: QEQueryLog: 7b2dcd92-e430-11ef-aed0-00000a000017 :0:json query is: { end_time: 1738808234780632 , enqueue_time: 1738808234795848 , query_metadata: {"enqueue_time": 1738808234795735} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:14:826.448 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.498 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.506 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.542 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.549 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.557 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.822 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346361100032, Pid 1]: [SYS_INFO]: QEQueryLog: 7b2dcd92-e430-11ef-aed0-00000a000017 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:14:826.898 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.928 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.938 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.947 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.962 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.969 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:826.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:14:826.999 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: 7b2dcd92-e430-11ef-aed0-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:14:827.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QEQueryLog: 7b2dcd92-e430-11ef-aed0-00000a000017 :0:json query is: { end_time: 1738808234780632 , enqueue_time: 1738808234795848 , query_metadata: {"enqueue_time": 1738808234795735} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:14:827.254 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-06 Thu 02:17:14:827.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-06 Thu 02:17:14:827.307 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346482292480, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-06 Thu 02:17:14:828.082 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346356901632, Pid 1]: [SYS_INFO]: QELog: Finished: QID 7b2dcd92-e430-11ef-aed0-00000a000017 Table ObjectRoutingInstance Time(ms) 31 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-06 Thu 02:17:14:828.105 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346457110272, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2696-2. table = ObjectRoutingInstance query_stats= [ qid = 7b2dcd92-e430-11ef-aed0-00000a000017 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941:ctest-vn22-78729941", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 29,0, 29,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 31 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-06 Thu 02:17:14:828.132 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346356901632, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-06 Thu 02:17:49:445.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346377893632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-06 Thu 02:18:49:446.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346369496832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-06 Thu 02:19:49:447.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346365298432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-06 Thu 02:20:49:447.560 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346365298432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-06 Thu 02:21:49:448.107 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346508343040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-06 Thu 02:22:49:448.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-06 Thu 02:23:32:597.067 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.23:33990 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:23:49:449.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346452911872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-06 Thu 02:24:49:449.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346377893632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-06 Thu 02:25:49:450.305 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346356901632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-06 Thu 02:26:00:327.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-2 [Thread 140346664601728, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.41:8091::10.0.0.41:59392 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256