2025-09-11 Thu 04:17:05:234.846 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"v0+nightly.r24.1.centos","build-time":"2025-09-11 00:09:15.426696","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10925-1.novalocal","build-id":"v0+nightly.r24.1.centos-nightly_r24.1_centos.el7@contrail","build-number":"unknown"}]} src/contrail-common/base/misc_utils.cc 41 2025-09-11 Thu 04:17:05:235.011 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: ROLE : Generator 2025-09-11 Thu 04:17:05:235.023 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: MODULE : contrail-dns 2025-09-11 Thu 04:17:05:235.030 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-4080-2 2025-09-11 Thu 04:17:05:235.035 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: NODE TYPE : Control 2025-09-11 Thu 04:17:05:235.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: INSTANCE ID : 0 2025-09-11 Thu 04:17:05:235.047 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: HTTP SERVER PORT : 8092 2025-09-11 Thu 04:17:05:235.059 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-09-11 Thu 04:17:05:235.344 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: HTTP Introspect Init 2025-09-11 Thu 04:17:05:235.481 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8092 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-11 Thu 04:17:05:235.545 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: Sandesh Http Server Port: 8092 2025-09-11 Thu 04:17:05:235.593 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: NOT Writing http_port 8092TO : /tmp/contrail-dns.27249.http_port 2025-09-11 Thu 04:17:05:235.603 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: CONNECT TO COLLECTOR: 1 2025-09-11 Thu 04:17:05:235.702 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: Idle 2025-09-11 Thu 04:17:05:235.892 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: No Client: 1757564225235809 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:05:236.033 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: Logging: DISABLED -> ENABLED 2025-09-11 Thu 04:17:05:236.049 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-09-11 Thu 04:17:05:236.133 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Processing scm::EvStart in state Idle 2025-09-11 Thu 04:17:05:236.183 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Disconnect 2025-09-11 Thu 04:17:05:251.393 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Processing scm::EvCollectorUpdate in state Disconnect 2025-09-11 Thu 04:17:05:270.373 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Connect : Start Connect timer 10.0.0.254:8086 2025-09-11 Thu 04:17:05:270.563 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:270.682 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: SANDESH: Send FAILED: 1757564225251314 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Disconnect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:05:270.710 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:05:270.722 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:270.800 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: SANDESH: Send FAILED: 1757564225261664 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Disconnect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:05:270.810 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:05:270.823 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:270.891 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: SANDESH: Send FAILED: 1757564225270547 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:05:270.900 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:05:484.402 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.20.0.254:8093 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-09-11 Thu 04:17:05:484.506 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: Bulk sync has been started controller/src/dns/main.cc 380 2025-09-11 Thu 04:17:05:484.572 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: Start RabbitMqReader and init Database src/contrail-common/config-client-mgr/config_client_manager.cc 295 2025-09-11 Thu 04:17:05:484.662 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection down src/contrail-common/config-client-mgr/config_amqp_client.cc 159 2025-09-11 Thu 04:17:05:484.674 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: Connect to Rabbit MQ with queue_delete TRUE src/contrail-common/config-client-mgr/config_amqp_client.cc 168 2025-09-11 Thu 04:17:05:484.816 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:484.851 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564225484645 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Down description = RabbitMQ connection down ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Database:Cassandra ], ] ] ] 2025-09-11 Thu 04:17:05:484.861 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:537.723 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection established src/contrail-common/config-client-mgr/config_amqp_client.cc 152 2025-09-11 Thu 04:17:05:537.765 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:537.827 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: SANDESH: Send FAILED: 1757564225537676 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Database:Cassandra ], ] ] ] 2025-09-11 Thu 04:17:05:537.838 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:537.777 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: Init Database src/contrail-common/config-client-mgr/config_client_manager.cc 300 2025-09-11 Thu 04:17:05:538.484 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Lost Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 578 2025-09-11 Thu 04:17:05:538.497 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db Init src/contrail-common/config-client-mgr/config_cassandra_client.cc 85 2025-09-11 Thu 04:17:05:538.509 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:538.562 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564225538450 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Down description = Lost Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector, Database:Cassandra connection down ], ] ] ] 2025-09-11 Thu 04:17:05:538.573 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:577.042 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db SetTableSpace src/contrail-common/config-client-mgr/config_cassandra_client.cc 93 2025-09-11 Thu 04:17:05:577.336 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db UseColumnFamily uuidTableName src/contrail-common/config-client-mgr/config_cassandra_client.cc 102 2025-09-11 Thu 04:17:05:577.370 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db UseColumnFamily fqnTableName src/contrail-common/config-client-mgr/config_cassandra_client.cc 108 2025-09-11 Thu 04:17:05:577.459 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:577.471 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: BulkDataSync Started src/contrail-common/config-client-mgr/config_cassandra_client.cc 391 2025-09-11 Thu 04:17:05:577.491 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:05:577.543 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: SANDESH: Send FAILED: 1757564225577429 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:05:577.565 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:05:585.330 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:588.168 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:592.169 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:598.484 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:601.479 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:605.677 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:610.758 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:614.137 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:624.368 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:627.528 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:636.327 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:637.394 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:639.738 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:653.717 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:655.693 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:659.609 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:665.806 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:675.948 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:682.807 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:694.595 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:700.960 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:710.779 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:714.149 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:715.754 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:716.726 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:726.599 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:729.815 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:732.297 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:738.359 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:742.744 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:750.066 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:751.875 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:753.749 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:755.413 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:765.765 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:766.901 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:768.198 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:773.669 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:774.908 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:776.305 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:787.156 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:789.632 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:791.000 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:794.884 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:796.681 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:798.298 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:801.394 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:804.035 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:805.215 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:810.132 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:811.087 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:816.929 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:826.387 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:855.080 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:858.818 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:882.133 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:884.697 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:887.328 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:900.722 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:901.788 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:905.371 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:934.703 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:936.046 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:939.470 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:943.486 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:946.189 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:951.602 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:958.434 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:959.578 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:961.163 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:980.046 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:983.990 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:05:987.396 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:008.148 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:010.232 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:012.832 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:034.534 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:041.789 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:043.136 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:049.838 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:051.538 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:052.613 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:055.591 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:056.545 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:059.481 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:072.288 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:073.630 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:075.239 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:082.380 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:091.373 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:093.373 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:103.745 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:110.743 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:112.572 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:137.664 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:139.162 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:141.722 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:254.707 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:256.341 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:257.676 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:280.794 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:285.787 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:290.894 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:303.263 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:318.675 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:332.004 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:367.891 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:376.008 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:380.890 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:393.838 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:395.323 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:406.495 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:415.527 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:429.871 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:432.079 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:439.518 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:440.654 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:444.629 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:458.069 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:460.005 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:461.770 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:463.281 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:464.508 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:465.658 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:466.766 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:466.960 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 556 2025-09-11 Thu 04:17:06:467.684 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862044825344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: BulkSyncDone by all readers src/contrail-common/config-client-mgr/config_cassandra_client.cc 552 2025-09-11 Thu 04:17:06:467.852 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862044825344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: End of RIB computed and notification sent src/contrail-common/config-client-mgr/config_client_manager.cc 191 2025-09-11 Thu 04:17:06:467.866 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: TRUE src/contrail-common/config-client-mgr/config_client_manager.cc 219 2025-09-11 Thu 04:17:06:468.670 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:default-virtual-network controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.719 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating project:default-domain:default-project controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.738 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:468.754 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:neutron_tag controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.768 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.788 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-system-config:default-global-system-config controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.807 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:468.830 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating application-policy-set:default-policy-management:default-application-policy-set controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.853 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating policy-management:default-policy-management controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.871 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating policy-management-application-policy-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:468.894 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-vrouter-config:default-global-system-config:default-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.910 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating application-policy-set-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:468.926 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-pending-cassandra-compaction-tasks controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:468.966 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:469.003 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:469.023 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating discovery-service-assignment:default-discovery-service-assignment controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:469.036 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-xmpp-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:469.056 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:469.501 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:471.190 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:06:471.243 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: SANDESH: Send FAILED: 1757564226467921 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:06:471.255 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:472.664 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:application controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:472.692 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:472.718 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:472.735 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:dci-network controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:472.780 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:472.795 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating api-access-list:default-global-system-config:default-api-access-list controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:472.812 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-api-access-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:472.825 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-conf-incorrect controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:472.844 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:472.862 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating security-group:default-domain:default-project:__no_rule__ controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:472.884 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-security-group controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:473.816 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:474.392 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:service-chain-flat-ipam controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.439 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:474.458 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:__link_local__ controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.501 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:474.524 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-prouter-tsn-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.556 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:474.583 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating domain:default-domain controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.601 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:tier controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.620 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:maintenance_mode_activate_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.651 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:474.667 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating namespace:default-domain:default-namespace controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.691 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-namespace controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:474.711 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:device_import_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:474.746 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:478.107 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:478.469 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-core-files controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:478.512 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:478.532 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:default-network-ipam controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:478.551 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:478.570 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-bottle-request-size-limit controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:478.587 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:478.615 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-project controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:478.629 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-address-mismatch-compute controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:478.647 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:479.612 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:480.769 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:cli_sync_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:480.819 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:480.844 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:fabric_config_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:480.878 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:480.901 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:netns-snat-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:480.927 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:480.941 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:container_cleanup_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:480.968 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:480.982 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:label controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.002 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-address-mismatch-control controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.031 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.048 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:maintenance_mode_deactivate_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.069 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.072 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862044825344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:481.086 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.726 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.747 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-xmpp-close-reason controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.769 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.785 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.823 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.839 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:neutron_fwaas controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.852 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:docker-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.873 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.887 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-bgp-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.906 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.920 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-appliance-set:default-global-system-config:native controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:481.968 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-service-appliance-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:481.984 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_os_computes_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.017 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.039 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-process-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.064 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.084 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:nat-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.109 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.124 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating application-policy-set:default-domain:default-project:default-application-policy-set controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.176 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-application-policy-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.193 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-table-limit-exceeded controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.211 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.224 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:hitless_upgrade_strategy_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.248 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.262 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:image_upgrade_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.292 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.309 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating fabric:default-global-system-config:default-fabric controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.334 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-fabric controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.350 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-appliance-set:default-global-system-config:opencontrail controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:482.371 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-service-appliance-set controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:482.989 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:483.915 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-storage-cluster-state controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:483.958 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:483.984 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_role_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.021 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.044 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-limit-exceeded controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.068 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.084 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:ztp_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.106 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.123 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-node-status controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.141 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.154 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:ip-fabric controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.187 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.219 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:deployment controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.234 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:analyzer-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.253 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.267 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-interface controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.292 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.305 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-disk-usage-high controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.321 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.333 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_device_template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.354 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.367 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-qos-config:default-global-system-config:default-global-qos-config controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.389 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-global-qos-config controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:484.404 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-process-status controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:484.429 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:486.831 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:06:486.993 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-partial-sysinfo controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.052 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.082 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:haproxy-loadbalancer-template controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.108 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.158 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-package-version-mismatch controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.187 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.216 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-disk-usage-critical controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.238 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.269 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-prouter-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.729 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.749 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-conf-elements-inconsistency controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.768 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.784 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:site controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.798 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.819 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.840 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.886 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.902 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:06:487.919 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:06:487.960 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: EndOfRib point was reached, IF-MAP objects have been created controller/src/dns/main.cc 86 2025-09-11 Thu 04:17:06:487.975 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_DNS.0, counting 0 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:487.995 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_DNS_record.0, counting 0 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:488.010 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.network_ipam.0, counting 2 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:488.026 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_network_network_ipam.0, counting 0 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:488.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.global_qos_config.0, counting 1 elements controller/src/dns/main.cc 98 2025-09-11 Thu 04:17:06:488.050 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: VnniConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 109 2025-09-11 Thu 04:17:06:488.060 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: IpamConfig has been updated after the bulk sync, the number of elements: 2 controller/src/dns/main.cc 120 2025-09-11 Thu 04:17:06:488.068 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: VirtualDnsConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 131 2025-09-11 Thu 04:17:06:488.076 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: VirtualDnsRecordConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 142 2025-09-11 Thu 04:17:06:488.084 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: GlobalQosConfig has been updated after the bulk sync controller/src/dns/main.cc 152 2025-09-11 Thu 04:17:06:488.146 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection down src/contrail-common/config-client-mgr/config_amqp_client.cc 159 2025-09-11 Thu 04:17:06:488.157 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: Connect to Rabbit MQ with queue_delete TRUE src/contrail-common/config-client-mgr/config_amqp_client.cc 168 2025-09-11 Thu 04:17:06:488.276 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:06:488.326 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564226488122 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Down description = RabbitMQ connection down ], ] ] flag_infos= [ [ ] ] description = Collector, Database:RabbitMQ connection down ], ] ] ] 2025-09-11 Thu 04:17:06:488.347 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:758.277 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection established src/contrail-common/config-client-mgr/config_amqp_client.cc 152 2025-09-11 Thu 04:17:06:758.300 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:06:758.360 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862019634944, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: TRUE src/contrail-common/config-client-mgr/config_client_manager.cc 219 2025-09-11 Thu 04:17:06:758.393 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:06:758.406 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:06:758.412 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564226758228 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:06:758.432 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:758.446 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:06:758.454 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:06:758.555 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: BIND named up; DNS is operational controller/src/dns/mgr/dns_mgr.cc 737 2025-09-11 Thu 04:17:06:758.603 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:06:758.647 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Idle 2025-09-11 Thu 04:17:06:758.721 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:06:758.875 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564226758545 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 306, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:06:758.900 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:06:758.911 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:06:758.934 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564226758573 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:06:758.942 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:06:758.951 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:06:758.997 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:06:759.144 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564226758708 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 308, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:06:759.162 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:759.083 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:07:759.510 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Connect : Start Connect timer 10.0.0.48:8086 2025-09-11 Thu 04:17:07:759.628 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:07:759.660 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: SANDESH: Send FAILED: 1757564227759321 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:07:759.670 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:07:759.679 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:07:759.785 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:07:759.830 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:07:759.853 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: SANDESH: Send FAILED: 1757564227759613 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 308, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:07:759.873 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:759.883 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:07:759.891 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:07:759.988 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:07:760.020 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Idle 2025-09-11 Thu 04:17:07:760.083 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:07:760.239 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: SANDESH: Send FAILED: 1757564227759946 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 308, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:07:760.257 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:760.268 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:07:760.291 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: SANDESH: Send FAILED: 1757564227759969 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:07:760.298 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:07:760.307 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:07:760.327 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:07:760.487 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: SANDESH: Send FAILED: 1757564227760073 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 309, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:07:760.503 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:07:834.755 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:07:975.190 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: DnsAgent [SYS_NOTICE]: DnsOperational: Bulk sync has been done (the DB has been synced with named) controller/src/dns/mgr/dns_mgr.cc 455 2025-09-11 Thu 04:17:09:760.360 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:09:760.658 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Connect : Start Connect timer 10.0.0.22:8086 2025-09-11 Thu 04:17:09:760.809 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:09:760.841 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564229760472 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:09:760.862 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:09:760.872 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:09:761.034 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564229760792 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 153 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 311, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:09:761.051 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:09:761.356 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:09:761.368 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:09:761.655 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:09:761.669 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:09:761.780 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:09:761.817 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Idle 2025-09-11 Thu 04:17:09:761.887 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:09:762.046 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564229761730 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 153 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 311, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:09:762.061 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:09:762.072 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:09:762.094 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564229761754 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:09:762.105 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:09:762.114 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:09:762.124 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:09:762.265 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564229761870 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 153 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 75 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 312, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:09:762.277 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:10:872.475 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:10:872.713 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:__no_rule__:ingress-access-control-list controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:10:872.762 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating security-group-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:10:925.213 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:10:925.391 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:__no_rule__:egress-access-control-list controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:10:925.441 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating security-group-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:10:994.971 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:017.171 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:017.290 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 2e1fc63d-6d97-43e5-861a-2c3093d27091 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:056.232 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:081.668 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:081.776 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance a79654b7-d507-44f7-bbb7-2b6bf4c79ce3 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:124.779 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:164.512 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:164.632 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 85522317-b3f9-447d-bea5-72a8afc7db3b controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:224.684 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:256.514 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:256.633 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 010c7f62-733c-4f41-aaaa-9712d470b6e4 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:11:384.129 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:384.335 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:ip-fabric:ip-fabric controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:11:384.381 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-network-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:11:422.223 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:11:445.706 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:022.283 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:082.072 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:114.230 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:131.168 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:171.820 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:174.125 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:216.845 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:216.982 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router ae44b01e-9ba6-46ee-b793-23f3d46999ec controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:219.233 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:219.342 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router d9886334-d660-45c3-97f9-3f7c9c1d2a94 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:323.148 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:323.253 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router 69d58baa-ec2b-4f35-bbd7-4ec0ee893627 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:330.746 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:330.853 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router d9886334-d660-45c3-97f9-3f7c9c1d2a94 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:399.718 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:399.835 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router 69d58baa-ec2b-4f35-bbd7-4ec0ee893627 controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:418.681 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:418.835 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: bgp_peering No entry in metadata map bgp_router ae44b01e-9ba6-46ee-b793-23f3d46999ec controller/src/ifmap/client/config_json_parser.cc 223 2025-09-11 Thu 04:17:13:456.046 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:472.722 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:511.984 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:13:761.937 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:13:762.199 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Connect : Start Connect timer 10.0.0.254:8086 2025-09-11 Thu 04:17:13:762.334 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:13:762.366 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564233762015 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:13:762.377 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:13:762.386 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:13:762.576 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564233762313 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 181 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 356, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:13:762.661 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:13:762.605 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:13:762.698 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:13:762.745 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:13:762.762 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:13:762.891 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:13:762.925 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Idle 2025-09-11 Thu 04:17:13:763.007 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:763.184 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SANDESH: Send FAILED: 1757564233762842 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 181 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 356, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:13:763.203 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:13:763.214 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:763.237 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SANDESH: Send FAILED: 1757564233762871 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:13:763.245 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:13:763.253 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:13:763.276 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:13:763.446 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SANDESH: Send FAILED: 1757564233762995 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 181 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 357, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:13:763.462 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:14:485.774 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:14:545.872 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:14:558.615 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:15:365.705 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:21:763.045 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:21:763.324 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Connect : Start Connect timer 10.0.0.48:8086 2025-09-11 Thu 04:17:21:763.470 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:21:763.514 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: SANDESH: Send FAILED: 1757564241763130 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:21:763.525 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:21:763.535 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:21:763.652 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:21:763.684 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:21:763.711 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: SANDESH: Send FAILED: 1757564241763451 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 361, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:21:763.733 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:21:763.744 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:21:763.751 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:21:763.858 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:21:763.891 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Idle 2025-09-11 Thu 04:17:21:763.960 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:21:764.127 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: SANDESH: Send FAILED: 1757564241763811 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 361, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:21:764.144 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:21:764.156 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:21:764.178 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: SANDESH: Send FAILED: 1757564241763832 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:21:764.185 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:21:764.193 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:21:764.214 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:21:764.375 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: SANDESH: Send FAILED: 1757564241763950 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:21:764.391 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:35:236.311 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:35:236.540 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564255236083 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 49 dequeues = 49 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564241763890 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564241764211 ] max_sm_queue_count = 4 ] 2025-09-11 Thu 04:17:35:236.562 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:35:236.574 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:35:236.748 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564255236260 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:35:236.775 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:37:764.122 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:17:37:764.693 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Connect : Start Connect timer 10.0.0.22:8086 2025-09-11 Thu 04:17:37:764.739 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:17:37:764.799 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:17:37:764.852 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:17:37:764.891 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564257764292 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:37:764.914 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:37:764.925 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:17:37:764.933 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:17:37:765.062 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:17:37:765.113 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Idle 2025-09-11 Thu 04:17:37:765.228 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:765.563 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564257764830 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:37:765.650 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:37:765.664 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:765.833 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564257765000 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 362, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:37:765.850 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:37:765.862 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:765.886 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564257765027 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:17:37:765.893 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:17:37:765.902 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:17:37:765.927 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:17:37:766.091 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564257765207 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 185 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 68 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 78 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 363, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:17:37:766.109 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:17:42:500.879 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:42:501.054 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:42:501.093 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:42:520.441 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:42:520.585 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:42:520.621 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:17:42:567.058 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:17:42:567.197 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:17:42:567.238 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:05:236.543 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:05:236.640 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564285236386 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 40 dequeues = 40 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 59 dequeues = 59 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564257765110 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564257765925 ] max_sm_queue_count = 5 ] 2025-09-11 Thu 04:18:05:236.668 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:05:236.679 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:05:236.847 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: SANDESH: Send FAILED: 1757564285236514 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 372, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:05:236.863 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:05:700.096 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:05:700.162 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: SANDESH: Send FAILED: 1757564285699993 [SYS_NOTICE]: UveDnsInfo: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 start_time = 1757564225484482 build_info = {"build-info":[{"build-version":"v0+nightly.r24.1.centos","build-time":"2025-09-11 00:09:15.426696","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10925-1.novalocal","build-id":"v0+nightly.r24.1.centos-nightly_r24.1_centos.el7@contrail","build-number":"unknown"}]} collector = 10.0.0.254:8086 self_ip_list= [ [ (*_iter12) = 10.20.0.254, ] ] ] 2025-09-11 Thu 04:18:05:700.179 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: UveDnsInfo 2025-09-11 Thu 04:18:07:765.207 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:18:07:765.487 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Connect : Start Connect timer 10.0.0.254:8086 2025-09-11 Thu 04:18:07:765.658 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:07:765.693 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: SANDESH: Send FAILED: 1757564287765305 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.254:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:07:765.705 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:18:07:765.715 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:07:765.907 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: SANDESH: Send FAILED: 1757564287765638 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.254:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 372, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:07:765.929 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:07:765.978 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-11 Thu 04:18:07:765.988 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: -(-1):EvTcpConnectFail 2025-09-11 Thu 04:18:07:766.052 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvTcpConnectFail in state Connect 2025-09-11 Thu 04:18:07:766.084 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Connect : EvTcpConnectFail 2025-09-11 Thu 04:18:07:766.231 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-09-11 Thu 04:18:07:766.275 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Idle 2025-09-11 Thu 04:18:07:766.352 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:07:766.553 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SANDESH: Send FAILED: 1757564287766177 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 372, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:07:766.572 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:07:766.583 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:07:766.607 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SANDESH: Send FAILED: 1757564287766207 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:07:766.614 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:18:07:766.623 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-11 Thu 04:18:07:766.642 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:07:766.809 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SANDESH: Send FAILED: 1757564287766339 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 188 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 373, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:07:766.824 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:10:053.075 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:10:053.271 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:10:053.315 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:10:080.957 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:10:081.105 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:10:081.145 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:10:185.321 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:10:185.547 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:10:185.633 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:24:766.224 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:24:766.406 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:24:766.465 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:25:199.847 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:25:199.988 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:25:200.025 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:25:249.844 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:25:249.998 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:25:250.037 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862036428544, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:583.567 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:34:583.730 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-3. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:34:583.768 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:627.108 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:34:627.272 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-2. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:34:627.321 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:34:658.041 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:18:34:658.191 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-ansible-os-4080-1. controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:18:34:658.232 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:18:35:236.784 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:35:236.869 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564315236631 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 48 dequeues = 48 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 70 dequeues = 70 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564287766274 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564287766640 ] max_sm_queue_count = 5 ] 2025-09-11 Thu 04:18:35:236.883 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:35:236.894 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Processing scm::EvSandeshSend in state Idle 2025-09-11 Thu 04:18:35:237.076 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: SANDESH: Send FAILED: 1757564315236754 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 400, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:35:237.096 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862028031744, Pid 27330]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:766.390 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-11 Thu 04:18:37:766.691 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: Connect : Start Connect timer 10.0.0.48:8086 2025-09-11 Thu 04:18:37:766.881 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:37:766.923 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: SANDESH: Send FAILED: 1757564317766496 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:37:766.943 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-11 Thu 04:18:37:766.953 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: Processing scm::EvSandeshSend in state Connect 2025-09-11 Thu 04:18:37:767.141 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: SANDESH: Send FAILED: 1757564317766857 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 400, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:37:767.159 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:769.877 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:48424::10.0.0.48:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-09-11 Thu 04:18:37:769.902 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: 10.0.0.22:48424::10.0.0.48:8086(-1) OnSessionEvent TCP Connected 2025-09-11 Thu 04:18:37:769.914 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: 10.0.0.22:48424::10.0.0.48:8086(-1):EvTcpConnected 2025-09-11 Thu 04:18:37:769.980 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvTcpConnected in state Connect 2025-09-11 Thu 04:18:37:770.016 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Connect : EvTcpConnected : Cancelling Connect timer 2025-09-11 Thu 04:18:37:770.069 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ClientInit 2025-09-11 Thu 04:18:37:770.091 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-4080-2:contrail-dns:0:Control count 1 2025-09-11 Thu 04:18:37:770.226 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state ClientInit 2025-09-11 Thu 04:18:37:770.237 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-09-11 Thu 04:18:37:770.263 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564317770060 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-09-11 Thu 04:18:37:770.272 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Received UVE message in wrong state : NodeStatusUVE 2025-09-11 Thu 04:18:37:770.281 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state ClientInit 2025-09-11 Thu 04:18:37:770.288 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-09-11 Thu 04:18:37:770.304 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Processing scm::EvSandeshSend in state ClientInit 2025-09-11 Thu 04:18:37:770.313 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:770.518 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: SANDESH: Send FAILED: 1757564317770210 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = ClientInit successful_connections = 1 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 401, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-11 Thu 04:18:37:770.543 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-09-11 Thu 04:18:37:772.578 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862049023744, Pid 27330]: OnMessage control in state: ClientInit 2025-09-11 Thu 04:18:37:772.642 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-09-11 Thu 04:18:37:772.677 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: ClientInit : EvSandeshMessageRecv 2025-09-11 Thu 04:18:37:772.733 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Received Ctrl Message with size 0 2025-09-11 Thu 04:18:37:772.760 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncUVE Syncing ObjectDns val name = cn-jenkins-deploy-platform-ansible-os-4080-2 start_time = 1757564225484482 build_info = {"build-info":[{"build-version":"v0+nightly.r24.1.centos","build-time":"2025-09-11 00:09:15.426696","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-10925-1.novalocal","build-id":"v0+nightly.r24.1.centos-nightly_r24.1_centos.el7@contrail","build-number":"unknown"}]} collector = 10.0.0.254:8086 self_ip_list= [ [ (*_iter12) = 10.20.0.254, ] ] proxy seq 1 2025-09-11 Thu 04:18:37:772.779 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncAllMaps for DnsState:0 period 0 cycle 0 without seqno , total = 1 2025-09-11 Thu 04:18:37:773.011 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-4080-2:Control:contrail-dns:0 client_info= [ status = ClientInit successful_connections = 1 pid = 27330 http_port = 8092 start_time = 1757564225235713 collector_name = collector_ip = 10.0.0.48:8086 collector_list= [ [ (*_iter40) = 10.0.0.254:8086, (*_iter40) = 10.0.0.48:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 48 dequeues = 48 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 70 dequeues = 70 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1757564287766274 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1757564287766640 ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 197 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 80 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 90 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 401, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 32 2025-09-11 Thu 04:18:37:773.040 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-09-11 Thu 04:18:37:773.066 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncUVE Syncing ObjectDns val name = cn-jenkins-deploy-platform-ansible-os-4080-2 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.48:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.254:9041, (*_iter6) = 10.0.0.22:9041, (*_iter6) = 10.0.0.48:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.254:5673, (*_iter6) = 10.0.0.22:5673, (*_iter6) = 10.0.0.48:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 22 2025-09-11 Thu 04:18:37:773.078 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-09-11 Thu 04:18:37:773.089 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncAllMaps for PeerFlapData:0 period 0 cycle 0 without seqno , total = 0 2025-09-11 Thu 04:18:37:773.096 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncAllMaps for PeerStatsData:0 period 0 cycle 0 without seqno , total = 0 2025-09-11 Thu 04:18:37:773.104 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: SyncAllMaps for XmppPeerInfoData:0 period 0 cycle 0 without seqno , total = 0 2025-09-11 Thu 04:18:37:773.138 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: Established 2025-09-11 Thu 04:20:11:139.027 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: XMPP [SYS_DEBUG]: XmppCreateConnection: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-2:10.20.0.25 SEND Xmpp creating dynamic channel 10.20.0.254:8093::10.20.0.25:43093 controller/src/xmpp/xmpp_server.cc 586 2025-09-11 Thu 04:20:11:139.171 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: XMPP [SYS_INFO]: XmppConnectionCreate: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-2:10.20.0.25 RECV Created Xmpp Server connection from cn-jenkins-deploy-platform-ansible-os-4080-2 To : controller/src/xmpp/xmpp_connection.cc 727 2025-09-11 Thu 04:20:11:139.266 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8093::10.20.0.25:43093 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:20:11:139.371 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-2:10.20.0.25 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.20.0.25 ( ) controller/src/xmpp/xmpp_state_machine.cc 1326 2025-09-11 Thu 04:20:11:139.445 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:139.599 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862032230144, Pid 27330]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = EvTcpPassiveOpen local_port = 8093 remote_port = 43093 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:158.630 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: XMPP [SYS_DEBUG]: XmppCreateConnection: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-2:10.20.0.193 SEND Xmpp creating dynamic channel 10.20.0.254:8093::10.20.0.193:38069 controller/src/xmpp/xmpp_server.cc 586 2025-09-11 Thu 04:20:11:158.694 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: XMPP [SYS_INFO]: XmppConnectionCreate: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-2:10.20.0.193 RECV Created Xmpp Server connection from cn-jenkins-deploy-platform-ansible-os-4080-2 To : controller/src/xmpp/xmpp_connection.cc 727 2025-09-11 Thu 04:20:11:159.982 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-ansible-os-4080-2:10.20.0.193 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.20.0.193 ( ) controller/src/xmpp/xmpp_state_machine.cc 1326 2025-09-11 Thu 04:20:11:160.085 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:160.176 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = EvTcpPassiveOpen local_port = 8093 remote_port = 38069 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:161.915 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8093::10.20.0.193:38069 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:20:11:177.741 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Established event = EvXmppOpen identifier = an-jenkins-deploy-platform-ansible-os-4080-2/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:11:360.673 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Established event = EvXmppOpen identifier = an-jenkins-deploy-platform-ansible-os-4080-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1710 2025-09-11 Thu 04:20:12:949.157 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:12:997.987 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:107.261 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:107.423 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-router:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-2 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:107.464 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862040626944, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-virtual-router controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:154.305 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862015436544, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:154.496 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-router:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-1 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:154.554 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862134773504, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-virtual-router controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:480.230 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:480.421 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-machine-interface:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-2:vhost0 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:480.464 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-router-virtual-machine-interface controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:480.500 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-machine-interface-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:542.471 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862053222144, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:595.534 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:13:595.745 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-machine-interface:default-global-system-config:an-jenkins-deploy-platform-ansible-os-4080-1:vhost0 controller/src/ifmap/ifmap_server_table.cc 95 2025-09-11 Thu 04:20:13:595.801 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-router-virtual-machine-interface controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:595.828 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862130575104, Pid 27330]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-machine-interface-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2025-09-11 Thu 04:20:13:660.047 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862126376704, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:20:14:011.294 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862023833344, Pid 27330]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 571 2025-09-11 Thu 04:21:35:541.400 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8092::10.20.0.14:56110 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:046.463 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8092::10.20.0.14:56114 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:158.902 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8092::10.20.0.254:50136 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:239.627 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8092::10.20.0.17:41520 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:347.802 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8092::10.20.0.254:50140 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-09-11 Thu 04:21:36:866.161 UTC cn-jenkins-deploy-platform-ansible-os-4080-2 [Thread 139862290700416, Pid 27330]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.254:8092::10.20.0.17:43670 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254