07/14/2025 00:40:04.449 7f7bfced35c8 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 07/14/2025 00:40:04.449 7f7bfced35c8 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 07/14/2025 00:40:04.449 7f7bfced35c8 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 07/14/2025 00:40:04.449 7f7bf45bf2e8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 07/14/2025 00:40:04.449 7f7bf45bf2e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 start_time = 1752453604449970 collector_name = collector_ip = collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 07/14/2025 00:40:04.450 7f7bf45bf2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.496 7f7bfced35c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.41:5673//', 'pyamqp://guest:guest@10.0.0.61:5673//', 'pyamqp://guest:guest@10.0.0.130:5673//'] 07/14/2025 00:40:04.505 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/14/2025 00:40:04.505 7f7bf440f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.505 7f7bf440f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:40:04.506 7f7bf440f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.506 7f7bf440f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.506 7f7bf440f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:40:04.506 7f7bf440f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.510 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/14/2025 00:40:04.510 7f7bf440f240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.510 7f7bf440f240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/14/2025 00:40:04.510 7f7bf440f240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.559 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/14/2025 00:40:04.560 7f7bf440f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.560 7f7bf440f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:40:04.560 7f7bf440f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.565 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/14/2025 00:40:04.565 7f7bf440f240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.566 7f7bf440f240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 07/14/2025 00:40:04.566 7f7bf440f240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.594 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 07/14/2025 00:40:04.595 7f7bfced35c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/14/2025 00:40:04.595 7f7bfced35c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/14/2025 00:40:04.595 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/14/2025 00:40:04.596 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 07/14/2025 00:40:04.596 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 07/14/2025 00:40:04.597 7f7bf6d5e978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 07/14/2025 00:40:04.597 7f7bf6d5e978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 07/14/2025 00:40:04.598 7f7bf43a4588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.598 7f7bf43a4588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:04.598 7f7bf43a4588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.600 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/14/2025 00:40:04.605 7f7bf9e496d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 07/14/2025 00:40:04.605 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 07/14/2025 00:40:04.606 7f7bf6d5e978 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 07/14/2025 00:40:04.643 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/14/2025 00:40:04.646 7f7bf4305390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.646 7f7bf4305390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 07/14/2025 00:40:04.646 7f7bf4305390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.654 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/14/2025 00:40:04.655 7f7bf4300198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:04.655 7f7bf4300198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:04.655 7f7bf4300198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:05.669 7f7bfced35c8 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 07/14/2025 00:40:05.676 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/14/2025 00:40:05.701 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/14/2025 00:40:07.091 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:40:08.450 7f7bf428c048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:08.452 7f7bf428c048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:08.452 7f7bf428c048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:08.452 7f7bf428c048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:08.452 7f7bf428c048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:08.452 7f7bf428c048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:08.453 7f7bf428c048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:09.093 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 07/14/2025 00:40:09.104 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:40:12.452 7f7bf4278a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:12.453 7f7bf4278a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:12.453 7f7bf4278a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:12.454 7f7bf4278a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:12.455 7f7bf4278a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:12.455 7f7bf4278a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:14.917 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:40:16.454 7f7bf423ceb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:16.456 7f7bf423ceb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:16.456 7f7bf423ceb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:16.456 7f7bf423ceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:16.456 7f7bf423ceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:16.456 7f7bf423ceb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:16.456 7f7bf423ceb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:16.457 7f7bf423ceb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:16.638 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:40:16.644 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:40:17.900 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:40:18.842 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 07/14/2025 00:40:18.847 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:40:20.060 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:40:20.456 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:20.458 7f7bf42b0cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:20.458 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:20.458 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:20.458 7f7bf42b0cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:20.458 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:20.458 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:20.459 7f7bf42b0cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:20.988 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:40:20.993 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:40:23.013 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:40:23.015 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 07/14/2025 00:40:23.019 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:40:24.459 7f7bf422f0f0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:24.461 7f7bf422f0f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:24.461 7f7bf422f0f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:24.461 7f7bf422f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:24.462 7f7bf422f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:24.462 7f7bf422f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:24.462 7f7bf422f0f0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:24.462 7f7bf422f0f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:24.462 7f7bf422f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:24.463 7f7bf422f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:40:24.463 7f7bf422f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:24.463 7f7bf422f0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:24.463 7f7bf422f0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:24.463 7f7bf422f0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:24.936 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:40:25.499 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:40:25.508 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:40:25.509 7f7bf42a6eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:25.509 7f7bf42a6eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:25.509 7f7bf42a6eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:25.527 7f7bfced35c8 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 07/14/2025 00:40:25.528 7f7bf41e8588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:25.528 7f7bf41e8588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 07/14/2025 00:40:25.528 7f7bf41e8588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:25.539 7f7bf41e62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:25.540 7f7bf41e62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:25.540 7f7bf41e62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:25.540 7f7bfced35c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 07/14/2025 00:40:25.540 7f7bfced35c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 07/14/2025 00:40:25.541 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 07/14/2025 00:40:25.542 7f7bfced35c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3634-3. 07/14/2025 00:40:25.551 7f7bf9e496d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 07/14/2025 00:40:25.747 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3634-3.'] 07/14/2025 00:40:25.791 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 07/14/2025 00:40:25.801 7f7bf9e496d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 07/14/2025 00:40:25.903 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3634-3.'] 07/14/2025 00:40:26.053 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Fetching all containers 07/14/2025 00:40:26.098 7f7bfced35c8 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 07/14/2025 00:40:26.306 7f7bfced35c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3634-3..config_queue 07/14/2025 00:40:26.307 7f7bfced35c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3634-3..tftp_queue 07/14/2025 00:40:26.808 7f7bfced35c8 [contrail-device-manager] [WARNING]: Process 139 prepared to run in Full mode... 07/14/2025 00:40:26.808 7f7bfced35c8 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 07/14/2025 00:40:26.808 7f7bfced35c8 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 07/14/2025 00:40:26.810 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 07/14/2025 00:40:26.811 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 07/14/2025 00:40:26.811 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 07/14/2025 00:40:26.811 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 07/14/2025 00:40:26.812 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 07/14/2025 00:40:26.812 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 07/14/2025 00:40:26.816 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 07/14/2025 00:40:26.818 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 07/14/2025 00:40:26.819 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 07/14/2025 00:40:26.819 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 07/14/2025 00:40:26.819 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 07/14/2025 00:40:26.819 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 07/14/2025 00:40:26.819 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 07/14/2025 00:40:26.819 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 07/14/2025 00:40:26.820 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 07/14/2025 00:40:26.821 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 07/14/2025 00:40:26.821 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 07/14/2025 00:40:26.821 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 07/14/2025 00:40:26.821 7f7bfced35c8 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 07/14/2025 00:40:26.827 7f7bfced35c8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 07/14/2025 00:40:26.899 7f7bf41e8cc0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 07/14/2025 00:40:26.900 7f7bf41e8cc0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 07/14/2025 00:40:26.900 7f7bef142438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:26.901 7f7bef142438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/14/2025 00:40:26.901 7f7bef142438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:26.913 7f7bef09a4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:26.914 7f7bef09a4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/14/2025 00:40:26.914 7f7bef09a4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:26.941 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 07/14/2025 00:40:26.947 7f7bf9e496d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 07/14/2025 00:40:27.218 7f7bf9e496d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3634-3.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3634-3..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3634-3..tftp_queue', 'device_manager_ztp.ztp_queue'] 07/14/2025 00:40:28.461 7f7bef027780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:28.463 7f7bef027780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:28.464 7f7bef027780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:28.464 7f7bef027780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/14/2025 00:40:28.464 7f7bef027780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:28.464 7f7bef027780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:28.464 7f7bef027780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:28.464 7f7bef027780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:32.464 7f7bf9e6d438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:32.466 7f7bf9e6d438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:32.466 7f7bf9e6d438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:32.466 7f7bf9e6d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:32.466 7f7bf9e6d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/14/2025 00:40:32.467 7f7bf9e6d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:32.467 7f7bf9e6d438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:32.467 7f7bf9e6d438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:32.467 7f7bf9e6d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:32.467 7f7bf9e6d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 07/14/2025 00:40:32.468 7f7bf9e6d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:32.468 7f7bf9e6d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:32.468 7f7bf9e6d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:32.468 7f7bf9e6d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:33.129 7f7beefaaac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:33.129 7f7beefaaac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:33.129 7f7beefaaac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:36.470 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:36.479 7f7beef39048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:36.479 7f7beef39048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:36.479 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:36.480 7f7beef39048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:40:36.480 7f7beef39048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:36.480 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:36.481 7f7beef39048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:36.481 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:36.481 7f7beef39048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:36.481 7f7beef39048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:36.481 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:36.482 7f7beef39048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:36.482 7f7beef39048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:40.479 7f7bf724a978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:40.481 7f7bf724a978 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:40.481 7f7bf724a978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:40.481 7f7bf724a978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:40.481 7f7bf724a978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:40:40.481 7f7bf724a978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:40.481 7f7bf724a978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:40.482 7f7bf724a978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:44.482 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:44.484 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:44.484 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:44.484 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:44.484 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:40:44.484 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:44.484 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:44.485 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:48.485 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:48.486 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:48.486 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:48.486 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:48.487 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:40:48.487 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:48.487 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:48.487 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:48.487 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:48.488 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:48.488 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:48.488 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:48.488 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:48.488 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:52.488 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:52.489 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:52.489 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:52.490 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:52.490 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:40:52.490 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:52.490 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:52.490 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:52.491 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:52.491 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:52.491 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:52.491 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:52.491 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:52.491 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:56.490 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:40:56.492 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:40:56.492 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:40:56.492 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:56.493 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:40:56.494 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:40:56.494 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:00.493 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:00.494 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:00.494 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:00.494 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:00.495 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:00.495 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:00.495 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:00.495 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:00.495 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:00.496 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:00.496 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:00.496 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:00.496 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:00.496 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:04.495 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:04.497 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:04.497 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:04.497 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:04.497 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:04.497 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:04.497 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:04.498 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:08.497 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:08.499 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:08.499 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:08.499 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:08.500 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:08.500 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:08.500 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:08.500 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:08.500 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:08.500 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:08.501 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:08.501 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:08.501 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:08.501 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:12.500 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:12.501 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:12.501 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:12.501 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:12.502 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:16.502 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:16.504 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:16.505 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:16.505 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:16.505 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:16.505 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:16.505 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:20.504 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:20.506 7f7bef084048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:20.506 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:20.506 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:20.507 7f7bef084048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:20.508 7f7bef084048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:24.507 7f7beef3e828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:24.507 7f7beef3e828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:24.508 7f7beef3e828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:24.509 7f7beef3e828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:24.509 7f7beef3e828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:24.509 7f7beef3e828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:24.509 7f7beef3e828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:24.509 7f7beef3e828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:28.510 7f7beef3e2e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:28.520 7f7beef3e2e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:28.520 7f7beef3e2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:28.520 7f7beef3e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:28.520 7f7beef3e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:28.520 7f7beef3e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:28.520 7f7beef3e2e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:28.521 7f7beef3e2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:32.520 7f7beef42240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:32.522 7f7beef42240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:32.522 7f7beef42240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:32.523 7f7beef42240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:32.523 7f7beef42240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:32.523 7f7beef42240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:32.523 7f7beef42240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:32.523 7f7beef42240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:32.524 7f7beef42240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:32.524 7f7beef42240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:32.524 7f7beef42240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:32.526 7f7beef42240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:32.527 7f7beef42240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:32.527 7f7beef42240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:36.522 7f7bf43da6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:36.523 7f7bf43da6d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:36.524 7f7bf43da6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:36.524 7f7bf43da6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:36.524 7f7bf43da6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:36.524 7f7bf43da6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:36.524 7f7bf43da6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:36.524 7f7bf43da6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:36.525 7f7bf43da6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:36.525 7f7bf43da6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:36.525 7f7bf43da6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:36.525 7f7bf43da6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:36.525 7f7bf43da6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:36.525 7f7bf43da6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:40.524 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:40.526 7f7beef39048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:40.526 7f7beef39048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:40.526 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:40.526 7f7beef39048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:40.526 7f7beef39048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:40.526 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:40.527 7f7beef39048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:44.526 7f7beef39b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:44.528 7f7beef39b70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:44.528 7f7beef39b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:44.528 7f7beef39b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:44.528 7f7beef39b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:44.528 7f7beef39b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:44.529 7f7beef39b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:48.529 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:48.530 7f7beef3b8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:48.530 7f7beef3b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:48.530 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:48.530 7f7beef3b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:48.530 7f7beef3b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:48.531 7f7beef3b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:52.531 7f7bef097a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:52.533 7f7bef097a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:52.534 7f7bef097a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:52.534 7f7bef097a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:52.534 7f7bef097a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:52.534 7f7bef097a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:52.534 7f7bef097a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:56.533 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:41:56.534 7f7beef3b8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:41:56.534 7f7beef3b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:41:56.535 7f7beef3b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:41:56.536 7f7beef3b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:00.536 7f7bef0e0240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:00.537 7f7bef0e0240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:00.537 7f7bef0e0240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:00.537 7f7bef0e0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:00.538 7f7bef0e0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:00.539 7f7bef0e0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.306 7f7bfced35c8 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 07/14/2025 00:42:01.307 7f7bfced35c8 [contrail-device-manager] [WARNING]: initializing cassandra cluster 07/14/2025 00:42:01.307 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 07/14/2025 00:42:01.308 7f7bfb24ed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.308 7f7bfb24ed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:42:01.308 7f7bfb24ed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.309 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 07/14/2025 00:42:01.338 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 07/14/2025 00:42:01.340 7f7beef95978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.340 7f7beef95978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 07/14/2025 00:42:01.341 7f7beef95978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.372 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 07/14/2025 00:42:01.372 7f7beef95780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.373 7f7beef95780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:42:01.373 7f7beef95780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:01.393 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:02.680 7f7bfced35c8 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 07/14/2025 00:42:02.693 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:02.702 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:02.970 7f7bfced35c8 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 07/14/2025 00:42:02.975 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:02.975 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 07/14/2025 00:42:03.038 7f7bfced35c8 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 07/14/2025 00:42:03.045 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:03.974 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 07/14/2025 00:42:03.979 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:03.986 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:04.538 7f7bef00cb70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:04.540 7f7bef00cb70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:04.540 7f7bef00cb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:04.540 7f7bef00cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:04.540 7f7bef00cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:42:04.540 7f7bef00cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:04.540 7f7bef00cb70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:04.541 7f7bef00cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:05.089 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:42:05.097 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:05.117 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:06.051 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 07/14/2025 00:42:06.056 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:06.062 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:07.001 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:42:07.007 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:07.015 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:07.976 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 07/14/2025 00:42:07.981 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:07.988 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:08.540 7f7bf50f8438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:08.542 7f7bf50f8438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:08.542 7f7bf50f8438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:08.542 7f7bf50f8438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:08.542 7f7bf50f8438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:42:08.542 7f7bf50f8438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:08.542 7f7bf50f8438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:08.543 7f7bf50f8438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:09.491 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:42:09.497 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:09.503 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:10.046 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 07/14/2025 00:42:10.052 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:10.058 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 07/14/2025 00:42:10.952 7f7bfced35c8 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 07/14/2025 00:42:10.957 7f7bfced35c8 [contrail-device-manager] [WARNING]: Zookeeper lock released. 07/14/2025 00:42:10.958 7f7beef3b6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:10.958 7f7beef3b6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:10.958 7f7beef3b6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:10.995 7f7bfced35c8 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.41:5673//', 'pyamqp://guest:guest@10.0.0.61:5673//', 'pyamqp://guest:guest@10.0.0.130:5673//'] 07/14/2025 00:42:10.995 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 07/14/2025 00:42:10.996 7f7beefeac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:10.996 7f7beefeac18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:42:10.996 7f7beefeac18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:10.996 7f7beefeac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:10.996 7f7beefeac18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:42:10.996 7f7beefeac18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.000 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 07/14/2025 00:42:11.000 7f7beeff1630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.001 7f7beeff1630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:11.001 7f7beeff1630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.032 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/14/2025 00:42:11.033 7f7beeff4588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.033 7f7beeff4588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:42:11.033 7f7beeff4588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.041 7f7bfced35c8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/14/2025 00:42:11.041 7f7beeff4588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.041 7f7beeff4588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:11.042 7f7beeff4588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.044 7f7beeff1f60 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 07/14/2025 00:42:11.045 7f7beefebc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.045 7f7beefebc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 07/14/2025 00:42:11.045 7f7beefebc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.050 7f7beeff1f60 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 07/14/2025 00:42:11.050 7f7beeff72e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:11.050 7f7beeff72e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:11.050 7f7beeff72e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:12.542 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:12.544 7f7bef00ab70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:12.544 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:12.544 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:12.544 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:42:12.544 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:12.544 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:12.545 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:16.545 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:16.547 7f7bef00ab70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:16.547 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:16.548 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:16.549 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:20.548 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:20.550 7f7bef00ab70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:20.551 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.61:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:20.552 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:20.552 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:20.552 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.61:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:20.552 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:24.551 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:24.552 7f7bef00ab70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 07/14/2025 00:42:24.553 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:24.553 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:24.553 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:42:24.553 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:24.553 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 07/14/2025 00:42:24.553 7f7bef00ab70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 07/14/2025 00:42:24.554 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:24.554 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.130:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 07/14/2025 00:42:24.554 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:24.554 7f7bef00ab70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:24.554 7f7bef00ab70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 139 http_port = 8096 start_time = 1752453604449970 collector_name = collector_ip = 10.0.0.130:8086 collector_list = [ 10.0.0.61:8086, 10.0.0.130:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 07/14/2025 00:42:24.554 7f7bef00ab70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 07/14/2025 00:42:28.074 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.130', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-3.'], 'id_perms': {'created': '2025-07-14T00:42:28.052936', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:28.052936', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9950577460056392259, 'uuid_mslong': 12908570115224127934}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b3247420-9428-49be-8a17-8d7ac0ab8243'}, 'oper': 'CREATE', 'request-id': 'req-58e2610d-8f8f-489a-8d12-8a8825e17ffe', 'type': 'analytics_node', 'uuid': 'b3247420-9428-49be-8a17-8d7ac0ab8243'} 07/14/2025 00:42:28.554 7f7bef005e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 07/14/2025 00:42:28.555 7f7bef005e10 [contrail-device-manager] [INFO]: Session Event: TCP Connected 07/14/2025 00:42:28.555 7f7bef005e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3634-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.61:5673, 10.0.0.130:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.61:9041, 10.0.0.130:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.61:2181, 10.0.0.130:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.61,10.0.0.130:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/14/2025 00:42:28.556 7f7bef005e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 07/14/2025 00:42:28.557 7f7beee29a20 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/14/2025 00:42:28.558 7f7beee29a20 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/14/2025 00:42:28.604 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.61', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-2.'], 'id_perms': {'created': '2025-07-14T00:42:28.596293', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:28.596293', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11883407136483060450, 'uuid_mslong': 4078582565323948468}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '389a093b-e13c-41b4-a4ea-57d4636a2ae2'}, 'oper': 'CREATE', 'request-id': 'req-bf9c71a4-72f1-4a50-9d18-041ec34842e8', 'type': 'analytics_node', 'uuid': '389a093b-e13c-41b4-a4ea-57d4636a2ae2'} 07/14/2025 00:42:28.688 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.41', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-1.'], 'id_perms': {'created': '2025-07-14T00:42:28.678326', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:28.678326', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12070768518932452244, 'uuid_mslong': 3454425749893237110}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2ff095ea-cd59-4976-a783-fc03a8efd394'}, 'oper': 'CREATE', 'request-id': 'req-e236fe76-a685-4e06-b5b8-149442ba2bc4', 'type': 'analytics_node', 'uuid': '2ff095ea-cd59-4976-a783-fc03a8efd394'} 07/14/2025 00:42:39.502 7f7beee29048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001016 07/14/2025 00:42:39.503 7f7beee29048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/14/2025 00:42:39.503 7f7beeff9828 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 07/14/2025 00:42:39.503 7f7beeff9828 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 07/14/2025 00:42:39.504 7f7beeff9828 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 07/14/2025 00:42:39.504 7f7beeff9828 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/14/2025 00:42:39.504 7f7beeff9828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 07/14/2025 00:42:39.504 7f7beee29978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 07/14/2025 00:42:39.506 7f7beee29978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 07/14/2025 00:42:44.887 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.61', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-2.'], 'id_perms': {'created': '2025-07-14T00:42:44.870523', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:44.870523', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12774525268743684561, 'uuid_mslong': 9420396815090730607}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '82bbf8fd-eeb4-466f-b148-3b0cb1b801d1'}, 'oper': 'CREATE', 'request-id': 'req-0f3ccdec-32ee-409d-bb25-a5932b61d64a', 'type': 'analytics_alarm_node', 'uuid': '82bbf8fd-eeb4-466f-b148-3b0cb1b801d1'} 07/14/2025 00:42:45.690 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.41', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-1.'], 'id_perms': {'created': '2025-07-14T00:42:45.681130', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:45.681130', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12203098766517191683, 'uuid_mslong': 4528685003801903590}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3ed91f04-5740-41e6-a95a-1dac6143c403'}, 'oper': 'CREATE', 'request-id': 'req-d2ab9650-3b82-444a-8e7c-e167b0baf269', 'type': 'analytics_alarm_node', 'uuid': '3ed91f04-5740-41e6-a95a-1dac6143c403'} 07/14/2025 00:42:45.708 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.130', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-3.'], 'id_perms': {'created': '2025-07-14T00:42:45.688040', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:45.688040', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11997995033472214073, 'uuid_mslong': 17851332151929291479}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f7bcac03-0341-42d7-a681-70ea33032439'}, 'oper': 'CREATE', 'request-id': 'req-d3beeaa4-0799-4ee7-8391-c619cf12f7cd', 'type': 'analytics_alarm_node', 'uuid': 'f7bcac03-0341-42d7-a681-70ea33032439'} 07/14/2025 00:42:49.991 7f7beee29048 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001296 07/14/2025 00:42:49.992 7f7beee29048 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 07/14/2025 00:42:49.992 7f7bef006390 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 07/14/2025 00:42:49.992 7f7bef006390 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 07/14/2025 00:42:49.993 7f7bef006390 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 07/14/2025 00:42:49.993 7f7bef006390 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 07/14/2025 00:42:56.870 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.130', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-3.'], 'id_perms': {'created': '2025-07-14T00:42:56.857075', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:56.857075', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13163976934955305852, 'uuid_mslong': 1219295110145590259}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '10ebce71-4c99-43f3-b6af-d746cf724b7c'}, 'oper': 'CREATE', 'request-id': 'req-cb443bc2-6b27-42a7-ac80-e8bc8d98806c', 'type': 'analytics_snmp_node', 'uuid': '10ebce71-4c99-43f3-b6af-d746cf724b7c'} 07/14/2025 00:42:56.906 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.61', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-2.'], 'id_perms': {'created': '2025-07-14T00:42:56.860306', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:56.860306', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12745860187249222928, 'uuid_mslong': 4518806071350084858}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3eb6062e-23b3-44fa-b0e2-644f51c8a510'}, 'oper': 'CREATE', 'request-id': 'req-f3cacff1-2514-4caf-9d87-2c5446964f77', 'type': 'analytics_snmp_node', 'uuid': '3eb6062e-23b3-44fa-b0e2-644f51c8a510'} 07/14/2025 00:42:56.955 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.41', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3634-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3634-1.'], 'id_perms': {'created': '2025-07-14T00:42:56.942986', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:42:56.942986', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11936719483675262773, 'uuid_mslong': 13593725920693797280}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'bca69db3-c4f9-41a0-a5a7-bf207ff2bf35'}, 'oper': 'CREATE', 'request-id': 'req-83469c10-32fe-4586-8e8c-2d2245d5652c', 'type': 'analytics_snmp_node', 'uuid': 'bca69db3-c4f9-41a0-a5a7-bf207ff2bf35'} 07/14/2025 00:44:37.766 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-4aa248b9-defe-47a7-8802-0bdb77ef144c', 'type': 'global_system_config', 'uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300'} 07/14/2025 00:44:37.831 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-fe244202-23e1-42c2-9c05-45d37db98157', 'type': 'global_system_config', 'uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300'} 07/14/2025 00:44:37.903 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3634-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-2'], 'id_perms': {'created': '2025-07-14T00:44:37.888900', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:44:37.888900', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13323314089344681767, 'uuid_mslong': 13562562462831888298}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'bc37e6b3-d124-47aa-b8e5-eb930658c727', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.192'}, 'oper': 'CREATE', 'request-id': 'req-aaa45b5d-2b59-407b-a93e-3a05ce379451', 'type': 'virtual_router', 'uuid': 'bc37e6b3-d124-47aa-b8e5-eb930658c727'} 07/14/2025 00:44:37.964 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3634-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-1'], 'id_perms': {'created': '2025-07-14T00:44:37.944993', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:44:37.944993', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9994481335382884935, 'uuid_mslong': 7969073564889269242}}, 'parent_type': 'global-system-config', 'parent_uuid': 'f8c5040d-b878-4c91-97a0-98ce8c3aa300', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6e97d62f-702e-43fa-8ab3-87d24dc4c247', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.18'}, 'oper': 'CREATE', 'request-id': 'req-1f864ecf-2021-4f7f-bf2e-719835844e6f', 'type': 'virtual_router', 'uuid': '6e97d62f-702e-43fa-8ab3-87d24dc4c247'} 07/14/2025 00:44:38.238 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-2', 'vhost0'], 'id_perms': {'created': '2025-07-14T00:44:38.225344', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:44:38.225344', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13053177758854541493, 'uuid_mslong': 9941745508634411568}}, 'parent_type': 'virtual-router', 'parent_uuid': 'bc37e6b3-d124-47aa-b8e5-eb930658c727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '89f82cdd-d17d-4230-b526-3403528df0b5', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:89:f8:2c:dd:d1']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '09504b60-e832-4023-a9e2-f11b7e1f273e'}]}, 'oper': 'CREATE', 'request-id': 'req-90f12ebc-1586-420a-a0b3-123cfe0b0440', 'type': 'virtual_machine_interface', 'uuid': '89f82cdd-d17d-4230-b526-3403528df0b5'} 07/14/2025 00:44:38.306 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-90f12ebc-1586-420a-a0b3-123cfe0b0440', 'type': 'virtual_machine_interface', 'uuid': '89f82cdd-d17d-4230-b526-3403528df0b5'} 07/14/2025 00:44:38.366 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-1', 'vhost0'], 'id_perms': {'created': '2025-07-14T00:44:38.336772', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-07-14T00:44:38.336772', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10217903660578305506, 'uuid_mslong': 18008357997113723658}}, 'parent_type': 'virtual-router', 'parent_uuid': '6e97d62f-702e-43fa-8ab3-87d24dc4c247', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f9ea8a2f-9e07-4b0a-8dcd-493e15eef5e2', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:f9:ea:8a:2f:9e']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '09504b60-e832-4023-a9e2-f11b7e1f273e'}]}, 'oper': 'CREATE', 'request-id': 'req-8f62c25d-64b4-4b1a-993b-e14da607ef29', 'type': 'virtual_machine_interface', 'uuid': 'f9ea8a2f-9e07-4b0a-8dcd-493e15eef5e2'} 07/14/2025 00:44:38.411 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3634-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-8f62c25d-64b4-4b1a-993b-e14da607ef29', 'type': 'virtual_machine_interface', 'uuid': 'f9ea8a2f-9e07-4b0a-8dcd-493e15eef5e2'} 07/14/2025 00:44:38.783 7f7beeff4f60 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-f5b58d6e-0d15-4a04-aae8-413ad179c3e2', 'type': 'access_control_list', 'uuid': '1d2d993f-319a-48af-8f3c-1cbaf2412a35'} 07/14/2025 00:44:53.097 7f7beeff9780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 07/14/2025 00:45:33.098 7f7beef92588 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 07/14/2025 01:37:17.940 7f7bf51d6588 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map