2024-10-21 Mon 02:00:41:102.229 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_INFO]: VersionInfoLog: {"build-info":[{"build-version":"nightly","build-time":"2024-10-21 00:11:14.032079","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-6068-1.novalocal","build-id":"nightly-nightly.el7","build-number":"@contrail"}]} src/contrail-common/base/misc_utils.cc 41 2024-10-21 Mon 02:00:41:102.607 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: ROLE : Generator 2024-10-21 Mon 02:00:41:102.627 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: MODULE : contrail-dns 2024-10-21 Mon 02:00:41:102.634 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: SOURCE : cn-jenkins-deploy-platform-juju-os-439-1 2024-10-21 Mon 02:00:41:102.641 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: NODE TYPE : Control 2024-10-21 Mon 02:00:41:102.646 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: INSTANCE ID : 0 2024-10-21 Mon 02:00:41:102.653 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: HTTP SERVER PORT : 8092 2024-10-21 Mon 02:00:41:102.665 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: System Log Send Rate Limit: 0 -> 0 2024-10-21 Mon 02:00:41:103.133 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: HTTP Introspect Init 2024-10-21 Mon 02:00:41:103.289 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8092 Initialization complete src/contrail-common/io/tcp_server.cc 137 2024-10-21 Mon 02:00:41:103.351 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: Sandesh Http Server Port: 8092 2024-10-21 Mon 02:00:41:103.413 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: NOT Writing http_port 8092TO : /tmp/contrail-dns.1053873.http_port 2024-10-21 Mon 02:00:41:103.424 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: CONNECT TO COLLECTOR: 1 2024-10-21 Mon 02:00:41:103.523 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: Idle 2024-10-21 Mon 02:00:41:103.740 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: No Client: 1729476041103638 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:00:41:104.102 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: Logging: DISABLED -> ENABLED 2024-10-21 Mon 02:00:41:104.125 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2024-10-21 Mon 02:00:41:104.544 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvStart in state Idle 2024-10-21 Mon 02:00:41:104.581 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Disconnect 2024-10-21 Mon 02:00:41:106.117 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvCollectorUpdate in state Disconnect 2024-10-21 Mon 02:00:41:106.897 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:00:41:109.138 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:00:41:114.373 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476041106098 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:00:41:114.435 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:00:41:114.454 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:00:41:114.537 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476041106694 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:00:41:114.549 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:00:41:114.558 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:00:41:114.634 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476041109105 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:00:41:114.644 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:00:41:382.985 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 10.0.0.30:8093 Initialization complete src/contrail-common/io/tcp_server.cc 137 2024-10-21 Mon 02:00:41:383.102 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: Bulk sync has been started controller/src/dns/main.cc 384 2024-10-21 Mon 02:00:41:383.185 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: Start RabbitMqReader and init Database src/contrail-common/config-client-mgr/config_client_manager.cc 303 2024-10-21 Mon 02:00:41:383.722 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection down src/contrail-common/config-client-mgr/config_amqp_client.cc 158 2024-10-21 Mon 02:00:41:383.736 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:00:41:383.757 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: Connect to Rabbit MQ with queue_delete TRUE src/contrail-common/config-client-mgr/config_amqp_client.cc 167 2024-10-21 Mon 02:00:41:383.816 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476041383284 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Down description = RabbitMQ connection down ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Database:Cassandra ], ] ] ] 2024-10-21 Mon 02:00:41:383.830 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:00:41:384.867 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:00:46:386.072 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:00:51:386.760 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:00:56:387.578 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:01:388.500 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:06:389.501 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:11:401.572 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:16:402.417 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:21:403.097 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:26:409.576 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:31:410.346 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:36:411.243 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:41:412.065 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_WARN]: ConfigClientMgrWarning: RabbitMQ SM: Caught exception while connecting to RabbitMQ: 10.0.0.30:5673 : Error in opening SSL/TLS connection for socket src/contrail-common/config-client-mgr/config_amqp_client.cc 220 2024-10-21 Mon 02:01:46:437.731 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection established src/contrail-common/config-client-mgr/config_amqp_client.cc 151 2024-10-21 Mon 02:01:46:437.782 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: Init Database src/contrail-common/config-client-mgr/config_client_manager.cc 307 2024-10-21 Mon 02:01:46:437.816 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Lost Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 580 2024-10-21 Mon 02:01:46:437.826 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db Init src/contrail-common/config-client-mgr/config_cassandra_client.cc 87 2024-10-21 Mon 02:01:46:437.745 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:46:439.407 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476106437664 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Database:Cassandra ], ] ] ] 2024-10-21 Mon 02:01:46:439.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:46:439.446 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:46:439.468 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476106437808 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Down description = Lost Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector, Database:Cassandra connection down ], ] ] ] 2024-10-21 Mon 02:01:46:439.475 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:46:555.803 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db SetTableSpace src/contrail-common/config-client-mgr/config_cassandra_client.cc 95 2024-10-21 Mon 02:01:46:556.590 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db UseColumnFamily uuidTableName src/contrail-common/config-client-mgr/config_cassandra_client.cc 104 2024-10-21 Mon 02:01:46:556.649 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Db UseColumnFamily fqnTableName src/contrail-common/config-client-mgr/config_cassandra_client.cc 110 2024-10-21 Mon 02:01:46:556.718 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:556.730 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: BulkDataSync Started src/contrail-common/config-client-mgr/config_cassandra_client.cc 393 2024-10-21 Mon 02:01:46:556.751 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:46:556.784 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476106556686 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:46:556.793 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:46:559.540 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:563.657 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:565.697 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:583.715 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:589.833 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:593.868 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:597.964 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:606.187 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:610.859 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:623.009 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:635.039 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:637.475 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:646.894 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:650.830 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:652.454 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:657.148 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:670.840 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:672.320 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:678.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:680.209 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:691.293 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:692.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:695.758 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:697.033 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:700.915 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:702.538 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:704.731 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:706.116 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:708.759 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:716.257 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:721.530 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:722.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:725.780 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:728.757 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:748.467 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:750.700 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:752.042 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:752.827 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:765.561 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:768.306 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:770.543 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:777.117 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:779.896 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:801.990 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:807.732 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:819.330 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:822.492 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:823.708 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:827.172 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:828.663 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:829.803 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:830.703 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:831.553 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:832.439 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:833.424 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:834.476 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:835.356 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:836.263 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:837.124 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:838.220 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:839.108 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:840.192 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:841.442 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:843.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:844.523 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:845.540 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:846.457 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:847.326 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:848.677 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:850.357 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:851.324 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:853.213 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:854.327 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:855.336 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:856.422 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:858.529 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:859.524 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:860.436 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:861.255 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:862.205 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:864.647 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:873.792 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:875.403 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:876.812 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:877.927 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:878.935 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:879.785 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:880.624 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:881.595 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:890.969 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:892.741 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:895.237 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:897.050 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:46:899.451 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:021.086 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:028.229 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:029.731 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:033.093 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:038.494 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:039.634 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:040.529 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:041.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:042.351 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:043.276 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:044.197 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:045.264 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:046.140 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:047.344 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:053.435 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:055.231 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:056.431 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:057.320 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:058.416 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:059.501 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:060.892 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:097.522 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:099.175 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:100.682 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:124.321 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.395 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.340 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.367 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.599 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.661 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.716 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: One reader finished BulkSync src/contrail-common/config-client-mgr/config_cassandra_client.cc 558 2024-10-21 Mon 02:01:47:124.774 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: BulkSyncDone by all readers src/contrail-common/config-client-mgr/config_cassandra_client.cc 554 2024-10-21 Mon 02:01:47:124.804 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Config Client Mgr SM: End of RIB computed and notification sent src/contrail-common/config-client-mgr/config_client_manager.cc 197 2024-10-21 Mon 02:01:47:124.894 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:47:124.954 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: SANDESH: Send FAILED: 1729476107124831 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:47:125.022 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:47:125.150 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: TRUE src/contrail-common/config-client-mgr/config_client_manager.cc 225 2024-10-21 Mon 02:01:47:131.041 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:131.419 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating namespace:default-domain:default-namespace controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:131.471 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating domain:default-domain controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:131.489 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-namespace controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:131.505 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating api-access-list:default-global-system-config:default-api-access-list controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:131.529 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-system-config:default-global-system-config controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:131.542 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-api-access-list controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:131.556 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:default-network-ipam controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:131.576 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating project:default-domain:default-project controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:131.588 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:132.423 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:133.295 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-project controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:133.362 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:service-chain-flat-ipam controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:133.393 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:133.412 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:tier controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:135.857 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:136.164 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:deployment controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:136.189 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating discovery-service-assignment:default-discovery-service-assignment controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:136.201 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating application-policy-set:default-domain:default-project:default-application-policy-set controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:136.223 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-application-policy-set controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:136.241 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:label controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:136.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:application controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:137.648 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:137.888 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-bottle-request-size-limit controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:137.920 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:137.938 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:__link_local__ controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:137.996 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:139.230 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:139.668 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:neutron_fwaas controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:139.711 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:ip-fabric controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:139.757 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:139.778 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:dci-network controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:139.825 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:139.854 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:default-virtual-network controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:139.876 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:147.407 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:147.551 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating security-group:default-domain:default-project:__no_rule__ controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:149.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-security-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:149.412 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating application-policy-set:default-policy-management:default-application-policy-set controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:149.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating policy-management:default-policy-management controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:149.463 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating policy-management-application-policy-set controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:149.484 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-vrouter-config:default-global-system-config:default-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:149.497 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating application-policy-set-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:149.546 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-global-vrouter-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:149.566 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:site controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:150.483 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:150.631 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating global-qos-config:default-global-system-config:default-global-qos-config controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:150.656 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-global-qos-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:153.406 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:47:153.745 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating fabric:default-global-system-config:default-fabric controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:153.797 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-fabric controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:47:153.827 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag-type:neutron_tag controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:47:156.675 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: EndOfRib point was reached, IF-MAP objects have been created controller/src/dns/main.cc 86 2024-10-21 Mon 02:01:47:156.730 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_DNS.0, counting 0 elements controller/src/dns/main.cc 98 2024-10-21 Mon 02:01:47:156.749 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_DNS_record.0, counting 0 elements controller/src/dns/main.cc 98 2024-10-21 Mon 02:01:47:156.766 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.network_ipam.0, counting 2 elements controller/src/dns/main.cc 98 2024-10-21 Mon 02:01:47:156.782 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.virtual_network_network_ipam.0, counting 0 elements controller/src/dns/main.cc 98 2024-10-21 Mon 02:01:47:156.798 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: The IF-MAP database table has been updated: __ifmap__.global_qos_config.0, counting 1 elements controller/src/dns/main.cc 98 2024-10-21 Mon 02:01:47:156.824 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: VnniConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 109 2024-10-21 Mon 02:01:47:156.839 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: IpamConfig has been updated after the bulk sync, the number of elements: 2 controller/src/dns/main.cc 120 2024-10-21 Mon 02:01:47:156.853 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: VirtualDnsConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 131 2024-10-21 Mon 02:01:47:156.868 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: VirtualDnsRecordConfig has been updated after the bulk sync, the number of elements: 0 controller/src/dns/main.cc 142 2024-10-21 Mon 02:01:47:156.881 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: GlobalQosConfig has been updated after the bulk sync controller/src/dns/main.cc 152 2024-10-21 Mon 02:01:47:156.988 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection down src/contrail-common/config-client-mgr/config_amqp_client.cc 158 2024-10-21 Mon 02:01:47:157.008 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: Connect to Rabbit MQ with queue_delete TRUE src/contrail-common/config-client-mgr/config_amqp_client.cc 167 2024-10-21 Mon 02:01:47:158.536 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:47:158.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476107156924 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Down description = RabbitMQ connection down ], ] ] flag_infos= [ [ ] ] description = Collector, Database:RabbitMQ connection down ], ] ] ] 2024-10-21 Mon 02:01:47:158.604 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:47:227.001 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:47:239.487 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: SANDESH: Send FAILED: 1729476107226343 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:47:239.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:47:227.044 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: RabbitMQ SM: RabbitMQ connection established src/contrail-common/config-client-mgr/config_amqp_client.cc 151 2024-10-21 Mon 02:01:47:239.834 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464027088640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: TRUE src/contrail-common/config-client-mgr/config_client_manager.cc 225 2024-10-21 Mon 02:01:47:243.175 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: -(-1) OnSessionEvent TCP Connect Failed 2024-10-21 Mon 02:01:47:243.202 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: -(-1):EvTcpConnectFail 2024-10-21 Mon 02:01:47:244.060 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvTcpConnectFail in state Connect 2024-10-21 Mon 02:01:47:244.083 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Connect : EvTcpConnectFail 2024-10-21 Mon 02:01:47:244.150 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 314 2024-10-21 Mon 02:01:47:244.189 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Idle 2024-10-21 Mon 02:01:47:244.312 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:47:244.341 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: SANDESH: Send FAILED: 1729476107244118 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:47:244.350 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:47:244.359 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:01:47:244.368 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:47:244.545 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: SANDESH: Send FAILED: 1729476107244298 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 224, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:01:47:244.567 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:47:246.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: BIND named up; DNS is operational controller/src/dns/mgr/dns_mgr.cc 737 2024-10-21 Mon 02:01:47:246.754 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:47:246.807 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476107246645 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476107244188 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476107244366 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:01:47:246.816 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:47:246.830 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:47:247.010 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476107246729 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 152 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 225, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:01:47:247.025 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:47:271.243 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:35814 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:01:47:276.409 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:35830 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:01:47:594.924 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:47:594.988 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: SANDESH: Send FAILED: 1729476107594807 [SYS_NOTICE]: UveDnsInfo: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 start_time = 1729476041383073 build_info = {"build-info":[{"build-version":"nightly","build-time":"2024-10-21 00:11:14.032079","build-user":"root","build-hostname":"jenkins-build-prepare-sandbox-6068-1.novalocal","build-id":"nightly-nightly.el7","build-number":"@contrail"}]} collector = 10.0.0.30:8086 self_ip_list= [ [ (*_iter12) = 10.0.0.30, ] ] ] 2024-10-21 Mon 02:01:47:595.003 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: UveDnsInfo 2024-10-21 Mon 02:01:47:652.550 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:35814 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:01:47:672.366 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:35830 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:01:48:038.338 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:48:038.514 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-node:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:48:038.648 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-node controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:48:059.047 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:48:059.222 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-database-node:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:48:059.262 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:48:247.678 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:01:48:248.897 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:01:48:249.101 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:48:249.139 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476108248696 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:48:249.157 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:48:249.172 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:48:249.373 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476108249081 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 29 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 235, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:01:48:249.391 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:48:463.035 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: DnsAgent [SYS_NOTICE]: DnsOperational: Bulk sync has been done (the DB has been synced with named) controller/src/dns/mgr/dns_mgr.cc 455 2024-10-21 Mon 02:01:48:690.550 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:48:690.881 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:image_upgrade_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:48:690.930 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:48:979.719 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:48:980.011 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:container_cleanup_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:48:980.074 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:49:213.887 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:49:214.145 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:cli_sync_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:49:214.193 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:49:288.909 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:49:289.112 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-appliance-set:default-global-system-config:opencontrail controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:49:289.155 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-service-appliance-set controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:49:444.105 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:49:444.429 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:hitless_upgrade_strategy_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:49:445.533 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:49:555.640 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:49:555.806 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-appliance-set:default-global-system-config:native controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:49:555.849 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-service-appliance-set controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:49:637.428 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:49:657.606 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:maintenance_mode_activate_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:49:657.668 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:49:865.153 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:49:865.405 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:maintenance_mode_deactivate_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:49:865.451 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:077.229 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:077.500 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_device_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:077.545 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:111.375 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:111.675 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:analyzer-template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:111.719 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:243.062 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:243.250 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:nat-template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:243.293 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:249.113 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:01:50:249.205 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:01:50:249.238 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:01:50:249.370 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Idle 2024-10-21 Mon 02:01:50:249.511 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:50:249.544 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476110249289 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:50:249.555 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:50:249.568 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:01:50:249.578 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:50:249.772 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476110249487 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 165 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 30 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 40 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 269, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:01:50:249.802 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:50:249.884 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > SSL Handshake failed due to error: 125 category: system message: Operation canceled src/contrail-common/io/ssl_server.cc 134 2024-10-21 Mon 02:01:50:303.670 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:303.960 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_os_computes_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:304.055 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:354.256 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:357.043 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:netns-snat-template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:357.106 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:485.199 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:485.429 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_role_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:485.479 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:506.005 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:506.187 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:haproxy-loadbalancer-template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:506.229 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:636.764 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:636.978 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating service-template:default-domain:docker-template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:638.421 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating domain-service-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:705.611 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:705.844 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:device_import_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:705.895 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:50:917.166 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:50:917.394 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:fabric_config_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:50:921.156 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:51:290.488 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:51:290.716 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:ztp_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:51:290.765 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:51:531.785 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:51:531.990 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:fabric_deletion_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:51:532.087 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:51:628.567 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:51:722.190 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:51:722.443 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:device_deletion_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:51:722.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:51:905.931 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:51:906.687 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:role_assignment_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:51:906.789 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:51:977.268 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:100.887 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:101.402 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:topology_discovery_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:101.467 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:249.508 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:01:52:249.809 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:01:52:249.938 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:52:249.985 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: SANDESH: Send FAILED: 1729476112249587 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:52:249.994 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:52:250.004 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:01:52:250.196 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: SANDESH: Send FAILED: 1729476112249921 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 179 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 52 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 308, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 19, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:01:52:250.219 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:52:309.469 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:309.930 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:fabric_onboard_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:310.020 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:484.220 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:484.459 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:search_ip_mac_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:484.509 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:549.324 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:549.538 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:__no_rule__:ingress-access-control-list controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:549.584 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating security-group-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:678.584 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:678.840 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:show_interface_details_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:678.897 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:727.884 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:728.055 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:__no_rule__:egress-access-control-list controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:728.097 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating security-group-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:846.491 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:846.714 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:show_config_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:52:846.766 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:52:861.542 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:913.131 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:52:913.278 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 1a05838d-2dd2-48a4-98d0-eafce7fcbc46 controller/src/ifmap/client/config_json_parser.cc 223 2024-10-21 Mon 02:01:53:053.535 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:053.764 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:test_overlay_connectivity_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:053.811 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:062.135 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:086.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:086.645 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 2eba0cc3-5657-4d99-ae8f-aae1bdcca0ef controller/src/ifmap/client/config_json_parser.cc 223 2024-10-21 Mon 02:01:53:165.860 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:246.740 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:247.708 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:show_ops_info_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:247.789 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:279.363 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:280.342 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 8e903556-5a15-4923-8cdb-758b07e9606f controller/src/ifmap/client/config_json_parser.cc 223 2024-10-21 Mon 02:01:53:378.181 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:493.857 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:494.110 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:check_multicast_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:494.167 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:512.924 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:513.095 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 04b5002c-41a0-4bf2-a878-358fac405bef controller/src/ifmap/client/config_json_parser.cc 223 2024-10-21 Mon 02:01:53:701.732 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:702.142 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:existing_fabric_onboard_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:702.213 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:775.092 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:775.345 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating access-control-list:default-domain:default-project:ip-fabric:ip-fabric controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:775.398 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-network-access-control-list controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:896.865 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:903.854 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:904.164 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:vcenter_import_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:905.656 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:954.644 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:53:954.849 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-alarm-node:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:53:954.895 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-alarm-node controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:53:968.788 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:081.578 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:081.809 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:discover_server_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:54:081.853 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:54:270.009 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:270.213 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:server_import_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:54:270.259 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:54:275.740 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:275.896 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating database-node:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:54:275.939 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-database-node controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:54:451.606 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:454.031 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:node_profile_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:54:454.112 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:54:624.211 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:624.435 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:show_mac_mob_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:54:624.504 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:54:802.805 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:54:803.076 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:rma_activate_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:54:803.123 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:011.842 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:012.156 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:show_chassis_info_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:012.228 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:176.732 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:177.047 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating job-template:default-global-system-config:hardware_inventory_template controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:178.303 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:310.465 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:310.861 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-properties:default-global-system-config:config_property controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:310.910 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-properties controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:420.688 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:421.306 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-management-ip controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:421.425 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:543.500 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:543.700 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-loopback-ip controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:543.742 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:670.147 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:670.317 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-overlay-loopback-ip controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:670.358 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:55:843.728 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:55:843.905 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-peer-ip controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:55:843.956 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:56:025.272 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:025.482 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-pnf-servicechain-ip controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:025.538 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:56:086.356 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:086.559 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating analytics-snmp-node:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:086.617 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-analytics-snmp-node controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:56:146.820 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:147.001 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-as-number controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:147.043 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:56:249.966 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:01:56:250.055 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:01:56:250.093 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:01:56:250.288 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Idle 2024-10-21 Mon 02:01:56:250.302 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > SSL Handshake failed due to error: 335544539 category: asio.ssl message: short read src/contrail-common/io/ssl_server.cc 134 2024-10-21 Mon 02:01:56:250.412 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:56:250.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476116250173 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:01:56:250.457 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:01:56:250.467 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:01:56:250.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:01:56:250.751 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476116250396 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 218 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 71 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 410, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:01:56:250.771 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:01:56:290.637 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:290.926 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating tag:label=fabric-ebgp-as-number controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:290.996 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating tag-tag-type controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:56:407.436 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:535.657 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:535.803 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48sh-afi controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:756.083 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:756.232 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48sh-afo controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:880.744 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:880.895 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48th-afi controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:56:997.656 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:56:997.796 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48th-afo controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:105.471 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:105.681 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48s-6q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:227.330 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:227.496 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-96s-8q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:340.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:340.659 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48t controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:473.067 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:473.207 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-48t-6q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:584.659 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:584.803 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-24q-2p controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:694.748 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:694.888 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100-24q-aa controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:810.976 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:811.115 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100e-48s-6q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:57:923.517 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:57:923.665 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100e-96s-8q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:038.289 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:039.200 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100e-48t-6q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:148.621 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:148.774 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5100e-24q-2p controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:289.313 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:289.470 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5110-48s-4c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:406.052 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:406.198 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5110-32q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:539.675 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:539.838 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5120-48y-8c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:652.836 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:653.086 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5120-32c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:773.199 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:773.377 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5120-48t-6c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:58:878.479 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:58:879.344 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5220-32cd controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:000.654 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:000.817 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5220-128c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:115.874 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:116.081 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5200-32c-32q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:224.890 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:225.094 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5200-48y controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:354.851 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:355.080 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx5210-64c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:466.979 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:467.146 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx10002-72q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:603.769 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:604.003 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx10002-36q controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:721.069 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:721.219 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx10002-60c controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:841.508 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:841.689 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx10016 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:864.347 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:864.516 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating config-node:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:01:59:864.559 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-config-node controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:01:59:958.404 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:01:59:958.547 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-qfx10008 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:069.870 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:070.132 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-vqfx-10000 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:180.818 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:181.269 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx80 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:250.428 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:02:00:250.910 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:02:00:251.090 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:00:251.136 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: SANDESH: Send FAILED: 1729476120250521 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:02:00:251.152 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:02:00:251.174 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:00:251.526 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: SANDESH: Send FAILED: 1729476120251066 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 252 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 114 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 479, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:00:251.551 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:00:302.676 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:302.869 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx80-48t controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:546.720 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:546.850 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx204 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:716.128 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:716.363 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx240 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:842.650 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:842.797 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx480 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:00:960.712 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:00:994.313 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx960 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:01:108.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:01:108.621 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx2008 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:01:226.975 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:01:227.157 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx2010 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:01:378.085 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:01:378.267 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx2020 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:01:547.240 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:01:547.376 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-mx10003 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:01:681.826 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:01:683.163 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-jnp10008 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:01:862.470 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:01:862.633 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-jnp10016 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:000.420 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:000.690 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-vmx controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:127.725 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:127.897 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx5400 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:253.763 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:254.014 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx5600 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:373.685 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:373.837 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx4600 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:501.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:501.749 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx4100 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:616.547 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:616.723 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx1500 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:737.098 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:737.240 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx240h-poe controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:02:850.319 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:02:850.484 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx5800 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:001.506 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:001.664 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-srx4200 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:107.684 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:107.880 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating hardware:juniper-vsrx controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:239.208 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:239.376 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:underlay-ip-clos controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:239.416 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:03:373.145 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:373.299 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:overlay-bgp controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:373.334 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:03:582.921 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:583.073 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:l2-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:583.111 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:03:721.514 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:721.670 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:l3-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:721.708 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:03:865.989 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:866.222 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:vn-interconnect controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:866.284 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:03:988.370 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:03:988.563 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:dc-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:03:988.612 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:109.260 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:109.501 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:assisted-replicator controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:109.547 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:247.927 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:248.091 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:port-profile controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:248.128 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:363.870 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:364.146 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:telemetry controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:364.183 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:493.808 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:493.979 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:firewall controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:494.020 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:624.854 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:625.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:infra-bms-access controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:625.292 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:766.495 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:766.671 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature:default-global-system-config:pnf-service-chaining controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:766.708 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:04:907.422 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:04:907.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating physical-role:default-global-system-config:leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:04:907.633 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:032.085 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:032.279 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating physical-role:default-global-system-config:spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:032.321 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:166.107 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:166.328 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating physical-role:default-global-system-config:superspine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:166.391 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:314.657 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:314.865 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating physical-role:default-global-system-config:pnf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:314.907 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:393.110 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:44750 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:02:05:449.642 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:44750 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:02:05:489.408 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:489.571 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:erb-ucast-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:489.612 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:619.779 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:619.953 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:crb-access controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:619.993 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:744.817 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:745.026 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:crb-mcast-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:745.069 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:05:907.721 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:05:908.673 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:crb-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:05:908.722 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:06:014.013 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:051.976 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:052.158 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:collapsed-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:06:052.205 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:06:179.243 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:331.465 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:331.666 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:dc-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:06:331.722 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:06:452.166 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:452.342 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:dci-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:06:452.384 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:06:546.962 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:44766 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:02:06:601.049 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:44766 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:02:06:622.528 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:622.664 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:ar-client controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:06:622.704 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:06:794.386 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:794.551 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:ar-replicator controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:06:794.604 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:06:949.110 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:06:949.250 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:route-reflector controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:06:949.289 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:078.695 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:078.850 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:lean controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:078.888 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:224.868 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:225.092 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating overlay-role:default-global-system-config:pnf-servicechain controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:225.237 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:227.539 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:227.759 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-conf-elements-inconsistency controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:227.811 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:275.551 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:275.774 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-address-mismatch-compute controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:275.845 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:374.827 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:375.025 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-address-mismatch-control controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:375.068 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:384.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:384.651 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:erb-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:384.689 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:451.603 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:451.835 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-bgp-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:451.893 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:518.388 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:518.555 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:crb-access-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:518.596 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:545.370 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:545.556 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-conf-incorrect controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:545.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:624.814 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:625.084 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-disk-usage-high controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:625.131 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:667.776 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:667.932 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:crb-access-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:667.982 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:07:735.652 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:07:735.837 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-disk-usage-critical controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:07:739.077 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:024.728 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:034.295 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:crb-gateway-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:034.348 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:062.423 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:062.638 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-node-status controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:062.685 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:144.775 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:145.016 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-partial-sysinfo controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:145.078 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:158.007 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:158.170 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:crb-mcast-gateway-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:158.232 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:235.991 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:236.198 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-process-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:236.245 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:252.603 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:02:08:254.028 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:02:08:254.048 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:02:08:254.741 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > SSL Handshake failed due to error: 335544539 category: asio.ssl message: short read src/contrail-common/io/ssl_server.cc 134 2024-10-21 Mon 02:02:08:254.808 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Idle 2024-10-21 Mon 02:02:08:254.981 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:02:08:255.026 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476128254582 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:02:08:255.057 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:02:08:255.073 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:02:08:255.131 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:02:08:255.508 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476128254959 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 318 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 116 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 178 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 657, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:08:255.541 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:08:298.227 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:298.978 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:collapsed-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:299.050 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:325.078 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:325.373 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-process-status controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:325.418 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:412.756 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:412.975 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-prouter-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:413.019 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:431.869 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:432.036 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:dc-gateway-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:432.085 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:541.601 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:541.820 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-prouter-tsn-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:541.866 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:567.531 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:567.707 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:dci-gateway-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:567.770 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:616.651 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:616.858 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-storage-cluster-state controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:616.902 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:694.438 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:694.680 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-interface controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:694.730 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:695.450 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:695.573 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:crb-gateway-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:695.611 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:758.334 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:758.522 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-xmpp-connectivity controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:758.567 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:819.925 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:820.116 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:crb-mcast-gateway-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:820.183 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:838.241 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:838.474 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-xmpp-close-reason controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:838.518 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:908.908 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:909.169 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-core-files controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:909.217 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:949.871 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:950.047 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:dc-gateway-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:950.098 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:08:978.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:08:978.393 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-pending-cassandra-compaction-tasks controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:08:978.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:047.814 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:048.061 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-package-version-mismatch controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:048.135 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:081.383 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:081.572 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:dci-gateway-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:081.638 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:126.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:126.820 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-limit-exceeded controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:126.865 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:203.588 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:203.812 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-table-limit-exceeded controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:203.867 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:226.299 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:226.451 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:ar-client-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:226.489 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:277.363 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:277.556 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating alarm:default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:277.602 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-alarm controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:358.741 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:358.980 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:ar-client-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:359.043 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:480.124 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:480.333 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:ar-replicator-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:480.389 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:602.517 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:602.699 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:ar-replicator-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:602.748 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:737.528 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:737.721 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:rr-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:737.799 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:868.923 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:869.105 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:rr-superspine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:869.153 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:09:991.389 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:09:991.562 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:rr-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:09:991.645 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:115.221 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:115.419 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:lean-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:115.460 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:259.224 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:259.498 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:lean-superspine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:259.552 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:427.033 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:427.402 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:pnf-servicechain-pnf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:427.626 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:479.503 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:557.060 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:557.235 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:pnf-servicechain-spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:557.276 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:670.839 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:671.061 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-definition:default-global-system-config:pnf-servicechain-leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:671.117 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-role-definition controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:797.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:797.711 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature-config:default-global-system-config:erb-leaf:l3-gateway controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:797.754 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:10:921.051 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:10:921.269 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature-config:default-global-system-config:ar-client-spine:assisted-replicator controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:10:921.335 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:032.062 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:032.224 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating feature-config:default-global-system-config:ar-client-leaf:assisted-replicator controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:032.265 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:162.260 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:162.643 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-qfx5k-lean controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:162.710 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:275.573 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:275.839 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-qfx5k controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:275.910 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:442.937 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:443.183 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-qfx10k-lean controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:443.250 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:544.084 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:569.367 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:569.557 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-qfx5120 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:569.610 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:706.865 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:707.059 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-qfx5220 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:707.112 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:843.017 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:843.198 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-qfx10k controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:843.250 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:11:995.640 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:11:995.974 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-mx controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:11:996.039 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:111.248 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:111.478 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:juniper-srx controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:111.538 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:238.013 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:238.265 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating node-profile:default-global-system-config:device-functional-group controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:238.310 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-node-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:357.828 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:358.009 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating telemetry-profile:default-domain:default-project:default-telemetry-profile-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:358.049 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-telemetry-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:489.818 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:490.085 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating telemetry-profile:default-domain:default-project:default-telemetry-profile-2 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:490.172 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-telemetry-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:607.333 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:607.491 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating telemetry-profile:default-domain:default-project:default-telemetry-profile-3 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:607.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-telemetry-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:656.041 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:773.727 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:773.909 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating telemetry-profile:default-domain:default-project:default-telemetry-profile-4 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:774.082 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-telemetry-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:12:880.103 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:12:881.554 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating telemetry-profile:default-domain:default-project:default-telemetry-profile-5 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:12:881.597 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-telemetry-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:006.923 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:007.105 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating telemetry-profile:default-domain:default-project:default-telemetry-profile-6 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:007.148 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-telemetry-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:117.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:117.851 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating sflow-profile:default-domain:default-project:sflow-all-interfaces controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:117.924 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-sflow-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:237.836 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:238.057 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating sflow-profile:default-domain:default-project:sflow-fabric-interfaces controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:238.124 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-sflow-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:357.721 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:357.973 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating sflow-profile:default-domain:default-project:sflow-access-interfaces controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:358.020 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-sflow-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:478.925 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:479.112 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating grpc-profile:default-domain:default-project:grpc-default-profile controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:479.153 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-grpc-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:602.250 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:603.406 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating snmp-profile:default-domain:default-project:snmp-default-profile controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:603.457 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-snmp-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:722.199 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:722.367 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating netconf-profile:default-domain:default-project:netconf-default-profile controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:722.410 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-netconf-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:842.490 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:842.685 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:L2-Server-Leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:842.730 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:13:957.399 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:13:957.565 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:L3-Server-Leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:13:957.609 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:081.650 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:081.899 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:L3-Gateway-Spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:082.155 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:212.587 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:212.762 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:L3-Storage-Leaf controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:212.806 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:344.995 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:345.170 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:L3-Server-Leaf-with-Optimized-Multicast controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:345.212 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:462.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:462.399 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:Centrally-Routed-Border-Spine controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:462.476 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:595.185 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:595.375 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:Centrally-Routed-Border-Spine-With-Optimized-Multicast controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:595.417 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:712.161 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:712.341 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:Border-Spine-in-Edge-Routed controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:712.383 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:829.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:829.502 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:Border-Leaf-in-Edge-Routed controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:829.565 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:14:955.685 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:14:955.862 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating device-functional-group:default-domain:default-project:Lean-Spine-with-Route-Reflector controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:14:955.903 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-device-functional-group controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:094.225 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:094.406 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-qfx5k-lean:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:094.488 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:211.462 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:211.656 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-qfx5k:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:211.728 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:328.927 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:329.143 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-qfx5120:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:329.210 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:447.659 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:447.848 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-qfx5220:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:447.891 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:563.986 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:564.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-qfx10k-lean:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:564.258 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:685.524 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:685.710 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-qfx10k:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:685.772 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:804.004 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:804.187 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-mx:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:804.254 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:15:931.769 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:15:932.490 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating role-config:default-global-system-config:juniper-srx:basic controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:15:932.554 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-role-config controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:16:088.462 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:16:088.689 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:16:251.207 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:16:251.386 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:16:254.951 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:02:16:255.249 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:02:16:255.383 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:16:255.417 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476136255034 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:02:16:255.427 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:02:16:255.436 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:16:255.644 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476136255364 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 397 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 192 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 252 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 886, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:16:255.686 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:16:406.551 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:16:406.757 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:16:579.213 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:16:579.428 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:16:741.610 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:16:741.800 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:16:908.562 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:16:908.835 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:17:074.587 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:17:074.832 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:17:235.646 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:17:235.867 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:17:247.038 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:17:247.186 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476137246885 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 29 dequeues = 29 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 43 dequeues = 43 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476136255042 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476136255435 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:02:17:247.210 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:17:247.221 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:17:247.470 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476137246996 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 403 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 198 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 252 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 898, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:17:247.495 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:17:395.695 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:17:396.031 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:17:548.614 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:17:548.820 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:17:707.436 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:17:707.659 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:17:869.444 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:17:869.726 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:18:036.443 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:18:036.622 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:18:197.766 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:18:198.002 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:18:356.166 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:18:356.398 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:18:530.257 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:18:530.472 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:18:722.795 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:18:722.992 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:18:895.559 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:18:895.827 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:051.345 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:051.633 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:215.456 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:215.687 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:374.198 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:374.467 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:524.426 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:525.175 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:676.621 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:676.821 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:827.313 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:827.503 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:19:997.922 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:19:998.129 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:20:155.648 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:20:155.839 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:20:306.418 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:20:306.604 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:20:460.752 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:20:460.952 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:20:616.729 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:20:616.921 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:20:780.084 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:20:780.359 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:20:943.193 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:20:943.612 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:21:109.452 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:21:109.650 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:21:273.147 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:21:273.323 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:21:436.261 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:21:436.442 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:21:644.661 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:21:644.867 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:21:804.749 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:21:806.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:21:984.179 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:21:984.406 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:22:141.809 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:22:142.091 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:22:303.700 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:22:303.950 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:22:459.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:22:459.779 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:22:614.362 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:22:614.572 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:22:792.591 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:22:792.813 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:22:944.468 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:22:944.636 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:23:117.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:23:117.445 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:23:291.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:23:291.406 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:23:442.713 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:23:442.898 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:23:604.405 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:23:604.607 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:23:776.626 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:23:777.280 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:23:936.000 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:23:936.203 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:24:099.714 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:24:099.965 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:24:253.015 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:24:253.202 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:24:425.350 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:24:425.536 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:24:586.201 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:24:586.501 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:24:729.262 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:24:729.446 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:24:891.841 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:24:892.098 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:25:055.252 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:25:055.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:25:219.398 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:25:219.635 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:25:383.950 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:25:384.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:25:531.444 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:25:531.615 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:25:688.894 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:25:689.117 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:25:843.991 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:25:844.197 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:26:012.912 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:26:013.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:26:185.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:26:185.806 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:26:358.818 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:26:359.037 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:26:523.200 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:26:523.403 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:26:688.239 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:26:688.507 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:26:841.959 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:26:842.166 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:013.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:013.644 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:182.597 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:182.797 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:334.719 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:334.907 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:503.000 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:503.177 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:665.743 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:666.061 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:823.594 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:823.775 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:27:991.566 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:27:991.759 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:28:138.392 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:28:138.600 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:28:302.925 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:28:303.143 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:28:458.194 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:28:458.393 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:28:617.317 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:28:617.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:28:778.218 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:28:778.399 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:28:946.072 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:28:946.266 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:29:112.423 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:29:112.670 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:29:250.181 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:29:250.390 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:29:416.426 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:29:416.615 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:29:587.613 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:29:587.802 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:29:753.881 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:29:754.124 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:29:917.569 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:29:917.753 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:30:089.149 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:30:089.342 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:30:248.251 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:30:248.428 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:30:406.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:30:406.827 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:30:561.001 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:30:561.216 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:30:723.369 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:30:723.639 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:30:887.733 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:30:887.922 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:31:067.156 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:31:067.357 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:31:230.767 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:31:231.894 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:31:402.295 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:31:402.509 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:31:576.367 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:31:576.558 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:31:736.809 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:31:737.020 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:31:918.586 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:31:918.821 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:32:112.774 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:32:113.043 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:32:255.370 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:02:32:255.437 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:02:32:255.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:02:32:258.727 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > SSL Handshake failed due to error: 335544539 category: asio.ssl message: short read src/contrail-common/io/ssl_server.cc 134 2024-10-21 Mon 02:02:32:258.808 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Idle 2024-10-21 Mon 02:02:32:259.081 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:02:32:259.123 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476152255489 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:02:32:259.133 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:02:32:259.144 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:02:32:259.240 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:02:32:259.468 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476152258925 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 494 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 289 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 252 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1081, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:32:259.498 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:32:271.330 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:32:271.532 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:32:428.593 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:32:591.478 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:32:591.671 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:32:751.292 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:32:751.522 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:32:916.896 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:32:917.136 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:33:088.805 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:33:089.156 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:33:257.467 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:33:257.675 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:33:411.216 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:33:411.403 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:33:574.032 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:33:574.219 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:33:741.304 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:33:741.475 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:33:898.243 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:33:898.471 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:34:061.178 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:34:061.466 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:34:225.718 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:34:226.412 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:34:389.146 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:34:389.341 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:34:538.309 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:34:538.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:34:697.131 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:34:697.317 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:34:850.339 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:34:850.531 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:014.468 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:014.770 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:180.381 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:180.666 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:339.417 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:339.611 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:502.214 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:502.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:661.144 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:661.399 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:814.775 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:814.984 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:35:995.992 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:35:996.242 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:36:150.524 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:36:150.713 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:36:308.178 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:36:308.412 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:36:466.633 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:36:466.820 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:36:624.423 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:36:624.638 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:36:789.474 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:36:790.050 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:36:948.084 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:36:948.257 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:37:127.822 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:37:128.010 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:37:301.232 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:37:301.466 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:37:454.209 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:37:454.565 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:37:612.778 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:37:622.355 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:37:786.502 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:37:786.735 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:37:945.970 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:37:946.198 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:38:128.370 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:38:128.586 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:38:272.904 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:38:273.616 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:38:437.171 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:38:437.411 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:38:619.714 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:38:619.930 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:38:762.053 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:38:762.263 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:38:926.240 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:38:926.469 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:39:089.165 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:39:089.454 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:39:258.487 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:39:258.698 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:39:420.454 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:39:420.757 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:39:577.127 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:39:577.315 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:39:749.216 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:39:749.416 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:39:910.456 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:39:910.850 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:40:073.084 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:40:073.301 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:40:241.397 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:40:241.595 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:40:400.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:40:400.497 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:40:562.908 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:40:563.530 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:40:720.946 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:40:721.188 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:40:874.676 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:40:874.902 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:035.717 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:035.929 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:201.381 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:201.627 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:355.309 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:355.514 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:518.256 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:518.555 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:667.168 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:667.528 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:824.240 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:824.444 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:41:995.867 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:41:996.132 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:42:153.211 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:42:153.444 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:42:370.700 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:42:370.918 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:42:493.277 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:42:493.498 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:42:651.183 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:42:651.520 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:42:825.336 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:42:825.558 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:42:986.577 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:42:987.485 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:43:138.124 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:43:138.318 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:43:305.490 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:43:305.721 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:43:470.545 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:43:472.492 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:43:635.412 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:43:635.640 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:43:802.212 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:43:802.474 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:43:985.113 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:43:985.319 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:44:134.314 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:44:134.509 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:44:317.222 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:44:317.442 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:44:482.244 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:44:482.502 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:44:644.724 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:44:644.922 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-feature controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:44:807.692 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:44:808.218 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:44:975.877 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:44:977.603 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:45:126.002 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:45:126.335 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:45:298.164 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:45:298.362 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:45:466.197 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:45:466.420 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:45:620.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:45:620.631 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating role-definition-overlay-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:45:777.561 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:45:777.748 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:45:931.327 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:45:931.509 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:46:092.178 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:46:092.441 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:46:267.815 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:46:268.062 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:46:417.389 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:46:417.611 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:46:577.685 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:46:577.903 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:46:735.537 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:46:735.729 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:46:897.179 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:46:897.381 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:47:051.796 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:47:052.038 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:47:243.913 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:47:244.220 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:47:247.359 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:02:47:247.477 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476167247114 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 49 dequeues = 49 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476152258804 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476152259192 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:02:47:247.514 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:47:247.531 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:02:47:247.901 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476167247308 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 587 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 381 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 252 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1266, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:47:247.950 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:47:413.499 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:47:413.794 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:47:574.763 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:47:575.116 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:47:731.334 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:47:731.616 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:47:925.956 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:47:926.166 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:48:112.659 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:48:112.848 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:48:258.929 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:02:48:259.309 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:02:48:259.468 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:48:259.505 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476168259025 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:02:48:259.529 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:02:48:259.539 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:48:259.752 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476168259444 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 592 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 386 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 252 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1276, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:02:48:259.771 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:02:48:285.403 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:48:285.579 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:48:663.277 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:48:663.527 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:48:844.207 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:48:844.425 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:48:992.449 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:48:992.686 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:49:150.467 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:49:150.727 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:49:314.792 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:49:315.279 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:49:478.348 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:49:478.617 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:49:665.385 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:49:665.586 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:49:807.570 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:49:807.756 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:49:963.965 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:49:965.013 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:139.177 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:139.360 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:301.607 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:302.178 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:344.018 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_DEBUG]: XmppCreateConnection: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 SEND Xmpp creating dynamic channel 10.0.0.30:8093::10.0.0.30:33333 controller/src/xmpp/xmpp_server.cc 586 2024-10-21 Mon 02:02:50:344.113 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_INFO]: XmppConnectionCreate: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Created Xmpp Server connection from cn-jenkins-deploy-platform-juju-os-439-1 To : controller/src/xmpp/xmpp_connection.cc 727 2024-10-21 Mon 02:02:50:344.159 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8093::10.0.0.30:33333 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:02:50:344.223 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.0.0.30 ( ) controller/src/xmpp/xmpp_state_machine.cc 1325 2024-10-21 Mon 02:02:50:344.292 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:50:344.317 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476170344185 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:50:344.327 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:50:344.341 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:50:344.358 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476170344283 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Active event = EvTcpPassiveOpen local_port = 8093 remote_port = 33333 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:50:344.366 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:50:349.391 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:50:349.429 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: SANDESH: Send FAILED: 1729476170349355 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Active next_state = Open Confirm event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-439-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:50:349.439 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:50:351.316 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Mode Server: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2024-10-21 Mon 02:02:50:351.319 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:50:351.401 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476170351287 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = OpenConfirm next_state = Trigger Server Ssl Handshake event = EvStartTls identifier = cn-jenkins-deploy-platform-juju-os-439-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:50:351.423 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:50:356.847 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_DEBUG]: XmppSslHandShakeMessage: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV success controller/src/xmpp/xmpp_connection.cc 402 2024-10-21 Mon 02:02:50:356.953 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:50:357.057 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476170356921 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = OpenConfirm next_state = OpenConfirm Feature Negotiation Success event = EvTlsHandShakeSuccess ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:50:357.069 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:50:359.248 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:366.293 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:50:366.350 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476170366248 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = OpenConfirm next_state = Established event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-439-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:50:366.365 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:50:464.707 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:464.912 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:504.780 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:505.044 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-router:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:50:505.104 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating global-system-config-virtual-router controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:637.703 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:637.886 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:853.186 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:853.483 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:926.912 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:50:927.129 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-machine-interface:default-global-system-config:cn-jenkins-deploy-platform-juju-os-439-1:vhost0 controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:50:927.202 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-router-virtual-machine-interface controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:50:927.232 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-machine-interface-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:51:032.705 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:032.854 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:080.088 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:080.301 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:51:238.792 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:238.993 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:51:410.536 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:410.827 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:51:567.601 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:567.810 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:51:729.435 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:729.642 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:51:899.393 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:51:900.307 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:046.123 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:047.715 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:204.034 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:204.247 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:361.157 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:361.405 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:525.923 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:526.170 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:688.804 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:689.082 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:845.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:846.092 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:52:996.755 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:52:996.993 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:53:151.948 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:53:154.416 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:53:317.025 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:53:317.216 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:53:488.482 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:53:489.118 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:53:658.131 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:53:659.206 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:53:797.215 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8093::10.0.0.30:33333 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:02:53:797.331 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Mode Server: Event: Tcp Connection Closed peer ip: 10.0.0.30 ( cn-jenkins-deploy-platform-juju-os-439-1/dns ) controller/src/xmpp/xmpp_state_machine.cc 1310 2024-10-21 Mon 02:02:53:797.495 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:53:797.519 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476173797418 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Established next_state = Idle event = EvTcpClose ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:53:797.528 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:53:797.538 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:53:797.550 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476173797430 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:53:797.565 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:53:797.565 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 Deleted Xmpp Server connection from cn-jenkins-deploy-platform-juju-os-439-1 To : cn-jenkins-deploy-platform-juju-os-439-1/dns controller/src/xmpp/xmpp_connection.cc 734 2024-10-21 Mon 02:02:53:815.181 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:53:815.436 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:53:982.104 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:53:996.709 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:54:159.477 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:54:159.739 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:54:329.069 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:54:329.295 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:54:489.173 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:54:489.399 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-hardware controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:54:641.775 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:54:641.891 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: physical_role Link name is empty node_profile 624902b2-87ef-4e3a-98c5-5a356ebf0788 controller/src/ifmap/client/config_json_parser.cc 217 2024-10-21 Mon 02:02:54:815.217 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:54:815.391 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: overlay_role Link name is empty node_profile 624902b2-87ef-4e3a-98c5-5a356ebf0788 controller/src/ifmap/client/config_json_parser.cc 217 2024-10-21 Mon 02:02:54:974.321 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:54:974.470 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: overlay_role Link name is empty node_profile 7d587b8b-0b07-4329-82e3-723977466375 controller/src/ifmap/client/config_json_parser.cc 217 2024-10-21 Mon 02:02:55:138.994 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:55:139.209 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:55:300.035 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:55:300.289 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:55:448.767 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:55:448.972 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:55:600.742 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:55:601.002 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:55:776.851 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:55:777.096 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:55:933.386 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:55:933.569 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:56:093.150 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:56:093.545 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:56:245.761 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:56:246.005 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating node-profile-job-template controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:56:414.285 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:56:414.536 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating telemetry-profile-sflow-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:56:569.143 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:56:569.327 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating telemetry-profile-sflow-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:56:732.082 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:56:732.303 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating telemetry-profile-sflow-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:56:903.807 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:56:904.008 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating telemetry-profile-grpc-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:57:064.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:57:064.649 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating telemetry-profile-snmp-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:57:229.442 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:57:229.654 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating telemetry-profile-netconf-profile controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:57:391.437 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:57:391.629 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:57:559.833 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:57:560.071 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:57:715.364 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:57:715.555 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:57:873.859 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:57:874.115 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:040.208 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:040.391 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:198.676 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:198.964 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:258.278 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_DEBUG]: XmppCreateConnection: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 SEND Xmpp creating dynamic channel 10.0.0.30:8093::10.0.0.30:34383 controller/src/xmpp/xmpp_server.cc 586 2024-10-21 Mon 02:02:58:258.333 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_INFO]: XmppConnectionCreate: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Created Xmpp Server connection from cn-jenkins-deploy-platform-juju-os-439-1 To : controller/src/xmpp/xmpp_connection.cc 727 2024-10-21 Mon 02:02:58:258.392 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8093::10.0.0.30:34383 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:02:58:258.445 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:58:258.501 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476178258413 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:58:258.516 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:58:258.451 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Mode Server: PassiveOpen in state: Idle peer ip: 10.0.0.30 ( ) controller/src/xmpp/xmpp_state_machine.cc 1325 2024-10-21 Mon 02:02:58:258.699 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:58:258.728 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: SANDESH: Send FAILED: 1729476178258687 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Active event = EvTcpPassiveOpen local_port = 8093 remote_port = 34383 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:58:258.739 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:58:266.608 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:58:266.653 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476178266575 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = Active next_state = Open Confirm event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-439-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:58:266.666 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:58:267.751 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:58:267.786 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: SANDESH: Send FAILED: 1729476178267730 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = OpenConfirm next_state = Trigger Server Ssl Handshake event = EvStartTls identifier = cn-jenkins-deploy-platform-juju-os-439-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:58:267.797 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:58:267.755 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: XMPP [SYS_INFO]: XmppUnconsumedEvent: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV Mode Server: Unconsumed Event xmsm::EvXmppKeepalive in state OpenConfirm controller/src/xmpp/xmpp_state_machine.cc 1649 2024-10-21 Mon 02:02:58:271.819 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: XMPP [SYS_DEBUG]: XmppSslHandShakeMessage: XMPP Peer cn-jenkins-deploy-platform-juju-os-439-1:10.0.0.30 RECV success controller/src/xmpp/xmpp_connection.cc 402 2024-10-21 Mon 02:02:58:271.984 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:58:272.035 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: SANDESH: Send FAILED: 1729476178271953 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = OpenConfirm next_state = OpenConfirm Feature Negotiation Success event = EvTlsHandShakeSuccess ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:58:272.051 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:58:275.007 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:02:58:275.058 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476178274965 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.30 state = OpenConfirm next_state = Established event = EvXmppOpen identifier = cn-jenkins-deploy-platform-juju-os-439-1/dns ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2024-10-21 Mon 02:02:58:275.074 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: XmppConnectionLog 2024-10-21 Mon 02:02:58:377.113 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:377.343 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:538.852 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:539.088 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:707.082 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:707.361 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:873.387 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:873.668 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating device-functional-group-physical-role controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:58:951.890 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:58:952.068 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating network-ipam:default-domain:default-project:_internal_ipam_ipv6_link_local controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:58:952.128 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:59:102.094 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:59:102.356 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network:default-domain:default-project:_internal_vn_ipv6_link_local controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:59:102.431 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating project-virtual-network controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:59:102.471 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapNodeOperation: Creating virtual-network-network-ipam:attr(default-domain:default-project:_internal_ipam_ipv6_link_local,default-domain:default-project:_internal_vn_ipv6_link_local) controller/src/ifmap/ifmap_server_table.cc 95 2024-10-21 Mon 02:02:59:102.500 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-network-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:59:102.525 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: IFMap [SYS_DEBUG]: IFMapLinkOperation: Creating virtual-network-network-ipam controller/src/ifmap/ifmap_server_table.cc 130 2024-10-21 Mon 02:02:59:138.572 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:59:651.985 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:59:678.172 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:02:59:678.296 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: IFMap [SYS_WARN]: ConfigurationMalformedReferenceWarningLog: instance_target No entry in metadata map routing_instance 16c39030-f601-4fb0-950d-22b731678844 controller/src/ifmap/client/config_json_parser.cc 223 2024-10-21 Mon 02:03:05:636.051 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:03:05:930.101 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:03:17:247.758 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:03:17:247.967 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476197247531 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 51 dequeues = 51 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 68 dequeues = 68 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476168259034 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476178274999 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:03:17:248.001 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:03:17:248.018 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:03:17:248.574 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476197247693 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1444, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:03:17:248.606 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:03:18:259.505 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463348643584, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:03:18:259.602 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:03:18:259.644 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:03:18:259.783 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Idle 2024-10-21 Mon 02:03:18:259.811 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > SSL Handshake failed due to error: 125 category: system message: Operation canceled src/contrail-common/io/ssl_server.cc 134 2024-10-21 Mon 02:03:18:259.963 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:03:18:260.801 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: SANDESH: Send FAILED: 1729476198259685 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:03:18:260.829 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:03:18:260.843 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:03:18:260.907 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:03:18:261.238 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: SANDESH: Send FAILED: 1729476198259943 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:03:18:261.269 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:03:47:248.121 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:03:47:248.217 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476227247832 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 55 dequeues = 55 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 74 dequeues = 74 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476198259780 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476198260898 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:03:47:248.231 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:03:47:248.241 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:03:47:248.545 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476227248070 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:03:47:249.112 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:03:48:259.954 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:03:48:260.215 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:03:48:260.447 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:03:48:260.496 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476228260037 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:03:48:260.506 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:03:48:260.516 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:03:48:260.849 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476228260423 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:03:48:260.875 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:04:17:248.441 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:04:17:248.573 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476257248259 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 59 dequeues = 59 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 79 dequeues = 79 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476228260046 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476228260514 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:04:17:248.588 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:04:17:248.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:04:17:248.912 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476257248395 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:04:17:248.933 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:04:18:260.398 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:04:18:260.530 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:04:18:260.576 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:04:18:260.706 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Idle 2024-10-21 Mon 02:04:18:261.017 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:04:18:261.076 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476258260616 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:04:18:261.087 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:04:18:261.098 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:04:18:261.135 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:04:18:261.480 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476258260862 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:04:18:261.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:04:47:248.710 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:04:47:248.820 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476287248531 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 63 dequeues = 63 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 85 dequeues = 85 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476258260704 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476258261133 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:04:47:248.848 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:04:47:248.857 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:04:47:249.173 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476287248654 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:04:47:249.202 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:04:48:260.901 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:04:48:261.328 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:04:48:261.554 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:04:48:261.604 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476288261086 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:04:48:261.618 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:04:48:261.633 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:04:48:262.197 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476288261517 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:04:48:262.251 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:17:249.015 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:05:17:249.156 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476317248848 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 67 dequeues = 67 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 90 dequeues = 90 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476288261099 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476288261630 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:05:17:249.185 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:17:249.195 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:05:17:249.497 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476317248954 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:05:17:249.518 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:18:261.487 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:05:18:261.556 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:05:18:261.567 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:05:18:261.736 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Idle 2024-10-21 Mon 02:05:18:261.871 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:05:18:261.906 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476318261601 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:05:18:261.917 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:05:18:261.940 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:05:18:261.974 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:05:18:262.270 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476318261843 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:05:18:262.292 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:47:249.304 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:05:47:249.410 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476347249094 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 71 dequeues = 71 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 96 dequeues = 96 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476318261734 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476318261972 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:05:47:249.427 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:47:249.438 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:05:47:249.744 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476347249235 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:05:47:249.773 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:48:261.900 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:05:48:262.156 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:05:48:262.315 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:05:48:262.356 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: SANDESH: Send FAILED: 1729476348261990 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:05:48:262.376 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:05:48:262.386 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:05:48:262.692 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: SANDESH: Send FAILED: 1729476348262287 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 668 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1445, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:05:48:262.716 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:05:51:727.896 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: ConfigClientManager [SYS_DEBUG]: ConfigClientMgrDebug: Cassandra SM: Established Cassandra connection src/contrail-common/config-client-mgr/config_cassandra_client.cc 573 2024-10-21 Mon 02:06:17:251.854 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:06:17:251.986 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: SANDESH: Send FAILED: 1729476377251709 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 75 dequeues = 75 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 101 dequeues = 101 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476348261999 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476348262385 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:06:17:252.001 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:06:17:252.012 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:06:17:252.300 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: SANDESH: Send FAILED: 1729476377251810 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1446, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:06:17:252.330 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:06:18:262.328 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:06:18:262.395 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:06:18:262.405 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:06:18:262.491 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Idle 2024-10-21 Mon 02:06:18:262.606 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:06:18:262.640 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476378262441 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:06:18:262.651 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:06:18:262.660 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:06:18:262.670 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:06:18:262.975 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476378262591 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1446, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:06:18:263.003 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:06:47:252.116 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:06:47:252.238 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476407251940 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 79 dequeues = 79 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 107 dequeues = 107 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476378262489 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476378262669 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:06:47:252.268 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:06:47:252.281 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:06:47:252.577 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476407252065 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1446, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:06:47:252.598 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:06:48:262.637 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:06:48:262.982 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:06:48:263.175 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:06:48:263.211 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476408262763 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:06:48:263.221 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:06:48:263.231 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:06:48:263.521 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476408263152 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1446, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 81, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:06:48:263.551 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:07:05:364.441 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:59328 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:07:05:410.888 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:59328 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:07:06:528.952 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:59336 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2024-10-21 Mon 02:07:06:575.350 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464231143552, Pid 1053912]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.30:8092::10.0.0.30:59336 < Read failed due to error asio.ssl 335544539 : short read src/contrail-common/io/tcp_session.cc 504 2024-10-21 Mon 02:07:17:252.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:07:17:252.572 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476437252261 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 112 dequeues = 112 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476408262772 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476408263229 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:07:17:252.587 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:07:17:252.597 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:07:17:252.904 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: SANDESH: Send FAILED: 1729476437252385 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:07:17:252.934 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464060675840, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:07:18:263.187 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:07:18:263.291 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:07:18:263.334 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:07:18:263.461 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Idle 2024-10-21 Mon 02:07:18:263.611 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:07:18:263.650 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476438263382 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:07:18:263.671 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:07:18:263.682 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:07:18:263.718 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:07:18:264.046 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476438263595 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 85, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:07:18:264.079 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:07:47:252.758 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:07:47:252.897 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: SANDESH: Send FAILED: 1729476467252565 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 118 dequeues = 118 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476438263458 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476438263716 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:07:47:252.913 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:07:47:252.924 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:07:47:253.268 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: SANDESH: Send FAILED: 1729476467252721 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 87, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:07:47:253.301 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464069072640, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:07:48:263.658 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:07:48:264.254 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:07:48:264.445 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:07:48:264.480 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476468263844 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:07:48:264.490 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:07:48:264.501 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:07:48:264.815 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476468264420 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 89, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:07:48:264.847 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:08:17:253.193 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:08:17:253.431 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476497252934 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 91 dequeues = 91 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 123 dequeues = 123 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476468263852 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476468264499 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:08:17:253.473 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:08:17:253.494 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:08:17:253.818 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476497253126 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 91, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:08:17:253.846 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:08:18:264.414 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464043882240, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:08:18:264.509 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:08:18:264.546 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:08:18:264.643 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Idle 2024-10-21 Mon 02:08:18:264.768 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:08:18:264.819 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476498264584 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:08:18:264.834 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:08:18:264.849 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:08:18:264.894 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:08:18:265.253 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: SANDESH: Send FAILED: 1729476498264752 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 93, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:08:18:265.292 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464052279040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:08:47:253.543 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:08:47:253.690 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: SANDESH: Send FAILED: 1729476527253290 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 95 dequeues = 95 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 129 dequeues = 129 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476498264641 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476498264892 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:08:47:253.723 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:08:47:253.737 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:08:47:254.255 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: SANDESH: Send FAILED: 1729476527253485 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 95, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:08:47:254.282 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464056477440, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:08:48:264.806 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:08:48:265.078 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:08:48:265.201 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:08:48:265.237 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476528264899 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:08:48:265.248 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:08:48:265.258 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:08:48:265.554 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: SANDESH: Send FAILED: 1729476528265185 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 97, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:08:48:265.589 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464073271040, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:09:17:253.877 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:09:17:254.125 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476557253680 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 99 dequeues = 99 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 134 dequeues = 134 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476528264907 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476528265256 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:09:17:254.150 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:09:17:254.167 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:09:17:254.722 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: SANDESH: Send FAILED: 1729476557253822 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 99, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:09:17:254.768 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:09:18:265.236 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:09:18:265.316 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:09:18:265.327 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:09:18:265.431 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Idle 2024-10-21 Mon 02:09:18:265.547 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:09:18:265.581 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476558265360 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:09:18:265.606 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:09:18:265.619 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:09:18:265.662 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:09:18:265.998 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476558265532 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 101, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:09:18:266.024 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:09:47:254.155 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:09:47:254.267 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476587253977 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 103 dequeues = 103 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 140 dequeues = 140 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476558265429 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476558265659 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:09:47:254.282 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:09:47:254.293 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:09:47:254.603 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: SANDESH: Send FAILED: 1729476587254108 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 103, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:09:47:254.636 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464064874240, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:09:48:265.603 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:09:48:265.911 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:09:48:266.078 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:09:48:266.125 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476588265697 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:09:48:266.136 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:09:48:266.145 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:09:48:266.451 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: SANDESH: Send FAILED: 1729476588266043 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 105, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:09:48:266.474 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463344445184, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:10:17:254.588 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:10:17:254.799 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476617254355 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 107 dequeues = 107 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 145 dequeues = 145 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476588265706 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476588266144 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:10:17:254.821 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:10:17:254.835 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:10:17:255.396 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476617254524 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 61 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 107, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:10:17:255.433 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:10:18:266.077 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464048080640, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:10:18:266.186 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:10:18:266.234 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:10:18:266.434 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Idle 2024-10-21 Mon 02:10:18:266.696 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:10:18:266.750 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: SANDESH: Send FAILED: 1729476618266277 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:10:18:266.777 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:10:18:266.793 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvTcpDeleteSession in state Idle 2024-10-21 Mon 02:10:18:266.853 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:10:18:267.445 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: SANDESH: Send FAILED: 1729476618266665 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 63 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 109, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:10:18:267.508 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464031287040, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:10:47:255.038 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:10:47:255.196 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476647254795 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 111 dequeues = 111 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 151 dequeues = 151 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1729476618266426 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1729476618266850 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:10:47:255.240 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:10:47:255.257 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:10:47:255.837 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: SANDESH: Send FAILED: 1729476647254972 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Idle successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 64 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 111, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:10:47:255.880 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463336048384, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:10:48:266.622 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvIdleHoldTimerExpired in state Idle 2024-10-21 Mon 02:10:48:266.985 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Connect : Start Connect timer 10.0.0.30:8086 2024-10-21 Mon 02:10:48:267.182 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:10:48:267.227 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: SANDESH: Send FAILED: 1729476648266746 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.30:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:10:48:267.243 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2024-10-21 Mon 02:10:48:267.258 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:10:48:267.828 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: SANDESH: Send FAILED: 1729476648267139 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 66 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 113, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:10:48:267.870 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464035485440, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:11:17:255.380 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:11:17:255.567 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476677255202 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 115 dequeues = 115 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 156 dequeues = 156 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Connect state_since = 1729476648266767 last_state = Connect last_event = scm::EvSandeshSend last_event_at = 1729476648267255 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 3 ] 2024-10-21 Mon 02:11:17:255.624 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:11:17:255.639 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Processing scm::EvSandeshSend in state Connect 2024-10-21 Mon 02:11:17:256.052 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: SANDESH: Send FAILED: 1729476677255325 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1:Control:contrail-dns:0 client_info= [ status = Connect successful_connections = 0 pid = 1053912 http_port = 8092 start_time = 1729476041103535 collector_name = collector_ip = 10.0.0.30:8086 collector_list= [ [ (*_iter40) = 10.0.0.30:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ConfigClientMgrDebug [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 669 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigClientMgrWarning [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = ConfigurationMalformedReferenceWarningLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = DnsOperational [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapLinkOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 454 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = IFMapNodeOperation [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 257 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 67 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveDnsInfo [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionCreate [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppCreateConnection [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppSslHandShakeMessage [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppUnconsumedEvent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1450, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 115, _iter59->first = sent _iter59->second = 0, ] ] ] 2024-10-21 Mon 02:11:17:256.083 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140463340246784, Pid 1053912]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2024-10-21 Mon 02:11:18:267.173 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464039683840, Pid 1053912]: 10.0.0.30:8086 EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:11:18:267.297 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvConnectTimerExpired in state Connect 2024-10-21 Mon 02:11:18:267.339 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Connect : EvConnectTimerExpired 2024-10-21 Mon 02:11:18:267.448 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Idle 2024-10-21 Mon 02:11:18:267.564 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Processing scm::EvSandeshSend in state Idle 2024-10-21 Mon 02:11:18:267.599 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: SANDESH: Send FAILED: 1729476678267379 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-os-439-1 process_status= [ [ [ module_id = contrail-dns instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvConnectTimerExpired ], [ type = Database name = Cassandra server_addrs= [ [ (*_iter6) = 10.0.0.30:9041, ] ] status = Up description = Established Cassandra connection ], [ type = Database name = RabbitMQ server_addrs= [ [ (*_iter6) = 10.0.0.30:5673, ] ] status = Up description = RabbitMQ connection established ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2024-10-21 Mon 02:11:18:267.610 UTC cn-jenkins-deploy-platform-juju-os-439-1 [Thread 140464077469440, Pid 1053912]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE