09/01/2025 02:50:36.991 7fe3e5973f88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 02:50:36.991 7fe3e5973f88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 02:50:36.991 7fe3e5973f88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 02:50:36.991 7fe3dd0552e8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 02:50:36.992 7fe3dd0552e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:50:36.992 7fe3dd0552e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 02:50:36.992 7fe3dd0552e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 02:50:36.992 7fe3dd0552e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:36.993 7fe3dd0552e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:50:36.993 7fe3dd0552e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:36.993 7fe3dd0552e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:36.993 7fe3dd0552e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1756695036992659 collector_name = collector_ip = collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 02:50:36.995 7fe3dd0552e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.051 7fe3e5973f88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.48:5673//'] 09/01/2025 02:50:37.064 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/01/2025 02:50:37.065 7fe3dcfa5588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.066 7fe3dcfa5588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:50:37.066 7fe3dcfa5588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.066 7fe3dcfa5588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.066 7fe3dcfa5588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:50:37.066 7fe3dcfa5588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.069 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 02:50:37.074 7fe3dcfa5240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.075 7fe3dcfa5240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:50:37.075 7fe3dcfa5240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.116 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/01/2025 02:50:37.116 7fe3dcfa50f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.116 7fe3dcfa50f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:50:37.116 7fe3dcfa50f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.119 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 02:50:37.120 7fe3eb857978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.120 7fe3eb857978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:50:37.121 7fe3eb857978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.134 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 09/01/2025 02:50:37.134 7fe3e5973f88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/01/2025 02:50:37.134 7fe3e5973f88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/01/2025 02:50:37.134 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 02:50:37.135 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 09/01/2025 02:50:37.135 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 02:50:37.135 7fe3df7ea978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 09/01/2025 02:50:37.135 7fe3df7ea978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 09/01/2025 02:50:37.135 7fe3dce44588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.136 7fe3dce44588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:37.136 7fe3dce44588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.138 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 02:50:37.140 7fe3e28dd6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 09/01/2025 02:50:37.140 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 09/01/2025 02:50:37.140 7fe3df7ea978 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 09/01/2025 02:50:37.169 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 02:50:37.171 7fe3dcd5d198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.171 7fe3dcd5d198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 02:50:37.171 7fe3dcd5d198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.177 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 02:50:37.177 7fe3dcddb780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:37.177 7fe3dcddb780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:37.177 7fe3dcddb780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:38.203 7fe3e5973f88 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 09/01/2025 02:50:39.213 7fe3e5973f88 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 09/01/2025 02:50:40.487 7fe3e5973f88 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 09/01/2025 02:50:40.495 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 02:50:40.525 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 02:50:40.992 7fe3dcd31a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:50:40.993 7fe3dcd31a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:50:40.993 7fe3dcd31a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:50:40.993 7fe3dcd31a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:50:40.994 7fe3dcd31a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:43.449 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:50:44.382 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 09/01/2025 02:50:44.386 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:50:44.994 7fe3dcd89240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:50:44.995 7fe3dcd89240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:50:44.995 7fe3dcd89240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:50:44.995 7fe3dcd89240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:44.995 7fe3dcd89240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:50:44.996 7fe3dcd89240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:46.513 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:50:48.219 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:50:48.224 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:50:48.997 7fe3dcd28f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:50:48.998 7fe3dcd28f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:48.999 7fe3dcd28f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:48.999 7fe3dcd28f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:48.999 7fe3dcd28f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:48.999 7fe3dcd28f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:50:48.999 7fe3dcd28f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:50.416 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:50:51.359 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 09/01/2025 02:50:51.363 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:50:52.998 7fe3dc4e2630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:50:52.999 7fe3dc4e2630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:50:52.999 7fe3dc4e2630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:50:53.000 7fe3dc4e2630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:53.454 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:50:54.347 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:50:54.352 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:50:56.403 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:50:56.406 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 09/01/2025 02:50:56.409 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:50:57.000 7fe3dc4c6828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:50:57.001 7fe3dc4c6828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:50:57.001 7fe3dc4c6828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:50:57.001 7fe3dc4c6828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:50:57.002 7fe3dc4c6828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:58.517 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:50:59.432 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:50:59.436 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:50:59.437 7fe3dcd48240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:59.437 7fe3dcd48240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:50:59.437 7fe3dcd48240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:59.447 7fe3e5973f88 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 09/01/2025 02:50:59.458 7fe3dc4f1f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:59.458 7fe3dc4f1f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 02:50:59.458 7fe3dc4f1f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:59.466 7fe3dc4f1a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:59.466 7fe3dc4f1a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:50:59.471 7fe3dc4f1a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:50:59.472 7fe3e5973f88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 09/01/2025 02:50:59.472 7fe3e5973f88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 09/01/2025 02:50:59.472 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 02:50:59.473 7fe3e5973f88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3996-2. 09/01/2025 02:50:59.475 7fe3e28dd6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 09/01/2025 02:50:59.555 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3996-2.'] 09/01/2025 02:50:59.570 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 02:50:59.574 7fe3e28dd6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 09/01/2025 02:50:59.603 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3996-2.'] 09/01/2025 02:50:59.982 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Fetching all containers 09/01/2025 02:51:00.020 7fe3e5973f88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 09/01/2025 02:51:00.183 7fe3e5973f88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3996-2..config_queue 09/01/2025 02:51:00.183 7fe3e5973f88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3996-2..tftp_queue 09/01/2025 02:51:00.635 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 02:51:00.638 7fe3e28dd6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 09/01/2025 02:51:00.684 7fe3e5973f88 [contrail-device-manager] [WARNING]: Process 138 prepared to run in Full mode... 09/01/2025 02:51:00.684 7fe3e5973f88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 09/01/2025 02:51:00.684 7fe3e5973f88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 09/01/2025 02:51:00.686 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 09/01/2025 02:51:00.687 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 09/01/2025 02:51:00.687 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 09/01/2025 02:51:00.687 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 09/01/2025 02:51:00.688 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 09/01/2025 02:51:00.688 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 09/01/2025 02:51:00.691 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 09/01/2025 02:51:00.692 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 09/01/2025 02:51:00.694 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 09/01/2025 02:51:00.695 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 09/01/2025 02:51:00.695 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 09/01/2025 02:51:00.695 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 09/01/2025 02:51:00.695 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 09/01/2025 02:51:00.695 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 09/01/2025 02:51:00.695 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 09/01/2025 02:51:00.696 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 09/01/2025 02:51:00.697 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 09/01/2025 02:51:00.697 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 09/01/2025 02:51:00.697 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 09/01/2025 02:51:00.697 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 09/01/2025 02:51:00.697 7fe3e5973f88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 09/01/2025 02:51:00.699 7fe3e5973f88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 09/01/2025 02:51:00.703 7fe3dc418b70 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 09/01/2025 02:51:00.703 7fe3dc418b70 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 09/01/2025 02:51:00.755 7fe3dc0cacc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:00.755 7fe3dc0cacc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/01/2025 02:51:00.755 7fe3dc0cacc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:00.766 7fe3dc4cc240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:00.766 7fe3dc4cc240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/01/2025 02:51:00.766 7fe3dc4cc240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:00.818 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3996-2.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3996-2..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3996-2..tftp_queue', 'device_manager_ztp.ztp_queue'] 09/01/2025 02:51:00.834 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 09/01/2025 02:51:00.841 7fe3e28dd6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 09/01/2025 02:51:00.869 7fe3e28dd6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3996-2.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3996-2..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3996-2..tftp_queue', 'device_manager_ztp.ztp_queue'] 09/01/2025 02:51:01.002 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:01.003 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:01.004 7fe3d7b1bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/01/2025 02:51:01.004 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:01.004 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:01.004 7fe3d7b1bb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:01.004 7fe3d7b1bb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:05.003 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:05.004 7fe3d7a52ac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:05.004 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:05.004 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:05.004 7fe3d7a52ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:05.005 7fe3d7a52ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:06.894 7fe3d7a52828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:06.895 7fe3d7a52828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:06.895 7fe3d7a52828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:09.005 7fe3d79c6198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:09.006 7fe3d79c6198 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:09.006 7fe3d79c6198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:09.006 7fe3d79c6198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:09.007 7fe3d79c6198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:13.007 7fe3d79f5c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:13.008 7fe3d79f5c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:13.008 7fe3d79f5c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:13.008 7fe3d79f5c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:13.009 7fe3d79f5c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:17.009 7fe3d7a59390 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:17.010 7fe3d7a59390 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:17.010 7fe3d7a59390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:17.010 7fe3d7a59390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:17.011 7fe3d7a59390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:21.011 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:21.013 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:21.013 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:21.013 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:21.013 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:21.013 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:21.013 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:21.014 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:25.013 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:25.014 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:25.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:25.016 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:25.016 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:25.016 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:29.015 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:29.016 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:29.016 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:29.016 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:29.016 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:29.016 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:29.016 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:29.017 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:33.017 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:33.019 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:33.019 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:33.019 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:33.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:37.020 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:37.021 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:37.021 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:37.021 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:37.021 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:37.021 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:37.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:41.022 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:41.023 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:41.023 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:41.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:41.025 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:41.025 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:45.024 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:45.025 7fe3d79c6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:45.025 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:45.025 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:45.025 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:45.026 7fe3d79c6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:49.027 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:49.028 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:49.029 7fe3d79d8f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:49.029 7fe3d79d8f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:49.029 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:49.029 7fe3d79d8f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:49.029 7fe3d79d8f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:53.028 7fe3d79c6588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:53.029 7fe3d79c6588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:53.029 7fe3d79c6588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:53.029 7fe3d79c6588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:53.029 7fe3d79c6588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:53.029 7fe3d79c6588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:53.029 7fe3d79c6588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:53.030 7fe3d79c6588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:57.030 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:51:57.031 7fe3d79d8f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:51:57.031 7fe3d79d8f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:51:57.031 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:57.031 7fe3d79d8f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:51:57.032 7fe3d79d8f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:01.032 7fe3e2aa4048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:01.033 7fe3e2aa4048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:01.033 7fe3e2aa4048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:01.033 7fe3e2aa4048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:01.033 7fe3e2aa4048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:01.033 7fe3e2aa4048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:01.033 7fe3e2aa4048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:01.034 7fe3e2aa4048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:05.034 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:05.035 7fe3dc4c9e10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:05.035 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:05.035 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:05.035 7fe3dc4c9e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:05.035 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:05.035 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:05.036 7fe3dc4c9e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:09.035 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:09.038 7fe3dc4bf6d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:09.038 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:09.038 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:09.038 7fe3dc4bf6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:09.038 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:09.039 7fe3dc4bf6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:13.038 7fe3d79d8438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:13.040 7fe3d79d8438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:13.040 7fe3d79d8438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:13.040 7fe3d79d8438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:13.040 7fe3d79d8438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:13.040 7fe3d79d8438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:13.040 7fe3d79d8438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:13.041 7fe3d79d8438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:17.040 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:17.042 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:17.043 7fe3d79c4eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:17.043 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:17.043 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:17.043 7fe3d79c4eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:17.043 7fe3d79c4eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.042 7fe3d79b7f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:21.043 7fe3d79b7f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:21.043 7fe3d79b7f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:21.043 7fe3d79b7f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:21.043 7fe3d79b7f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:21.043 7fe3d79b7f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:21.043 7fe3d79b7f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:21.044 7fe3d79b7f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.924 7fe3e5973f88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 09/01/2025 02:52:21.924 7fe3e5973f88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 09/01/2025 02:52:21.924 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 02:52:21.924 7fe3e3c0b588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.925 7fe3e3c0b588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:52:21.925 7fe3e3c0b588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:21.925 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 02:52:22.013 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 02:52:22.014 7fe3d7af5978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:22.014 7fe3d7af5978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 02:52:22.014 7fe3d7af5978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:22.020 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 02:52:22.020 7fe3d7af5828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:22.020 7fe3d7af5828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:52:22.020 7fe3d7af5828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:22.032 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:22.510 7fe3e5973f88 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 09/01/2025 02:52:22.513 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:22.518 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:23.375 7fe3e5973f88 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 09/01/2025 02:52:23.380 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:23.380 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 02:52:23.457 7fe3e5973f88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 02:52:23.468 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:24.603 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 09/01/2025 02:52:24.609 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:24.620 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:25.044 7fe3d7b112e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:25.045 7fe3d7b112e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:25.045 7fe3d7b112e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:25.045 7fe3d7b112e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:25.045 7fe3d7b112e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:52:25.045 7fe3d7b112e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:25.045 7fe3d7b112e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:25.046 7fe3d7b112e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:25.519 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:52:25.523 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:25.528 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:26.533 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 09/01/2025 02:52:26.536 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:26.542 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:28.259 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:52:28.266 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:28.271 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:29.046 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:29.047 7fe3d7a0f6d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:29.047 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:29.047 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:29.047 7fe3d7a0f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:52:29.047 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:29.047 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:29.048 7fe3d7a0f6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:29.392 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 09/01/2025 02:52:29.397 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:29.402 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:30.548 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:52:30.552 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:30.557 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:31.486 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 09/01/2025 02:52:31.489 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:31.494 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 09/01/2025 02:52:32.584 7fe3e5973f88 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 09/01/2025 02:52:32.588 7fe3e5973f88 [contrail-device-manager] [WARNING]: Zookeeper lock released. 09/01/2025 02:52:32.589 7fe3d7af50f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.589 7fe3d7af50f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:32.589 7fe3d7af50f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.632 7fe3e5973f88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.48:5673//'] 09/01/2025 02:52:32.632 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 09/01/2025 02:52:32.633 7fe3d7a0b588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.633 7fe3d7a0b588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:52:32.633 7fe3d7a0b588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.633 7fe3d7a0b588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.633 7fe3d7a0b588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:52:32.633 7fe3d7a0b588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.640 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 02:52:32.640 7fe3d7a0b048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.640 7fe3d7a0b048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:32.640 7fe3d7a0b048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.671 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/01/2025 02:52:32.672 7fe3d7a0b048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.672 7fe3d7a0b048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:52:32.672 7fe3d7a0b048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.676 7fe3e5973f88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 02:52:32.676 7fe3d7a11978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.676 7fe3d7a11978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:32.676 7fe3d7a11978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.677 7fe3d7a16198 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 09/01/2025 02:52:32.678 7fe3d7a15438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.678 7fe3d7a15438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:52:32.678 7fe3d7a15438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.681 7fe3d7a16198 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 02:52:32.681 7fe3dc0b2198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:32.681 7fe3dc0b2198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:32.681 7fe3dc0b2198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:33.047 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:33.049 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:33.050 7fe3d799b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:33.050 7fe3d799b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:33.050 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:33.050 7fe3d799b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:33.050 7fe3d799b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:37.050 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:52:37.051 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:37.052 7fe3d799b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:52:37.052 7fe3d799b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:37.052 7fe3d799b8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:37.052 7fe3d799b8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 http_port = 8096 start_time = 1756695036992659 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:52:37.052 7fe3d799b8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:52:40.261 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-2.'], 'id_perms': {'created': '2025-09-01T02:52:40.246068', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:52:40.246068', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11837862699443603904, 'uuid_mslong': 3643031863884924645}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '328ea62d-b84a-46e5-a448-8967cc2ec9c0'}, 'oper': 'CREATE', 'request-id': 'req-9d1cfb99-e99e-4a9d-a8d4-c6de153b5d9f', 'type': 'analytics_node', 'uuid': '328ea62d-b84a-46e5-a448-8967cc2ec9c0'} 09/01/2025 02:52:40.296 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-3.'], 'id_perms': {'created': '2025-09-01T02:52:40.283042', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:52:40.283042', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11071398211345259853, 'uuid_mslong': 14247965871800404244}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c5baf178-b08d-4914-99a5-8203b2211d4d'}, 'oper': 'CREATE', 'request-id': 'req-8f87cdaf-4fbd-41e8-9d6a-09a449ec724e', 'type': 'analytics_node', 'uuid': 'c5baf178-b08d-4914-99a5-8203b2211d4d'} 09/01/2025 02:52:40.337 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-1.'], 'id_perms': {'created': '2025-09-01T02:52:40.323374', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:52:40.323374', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10532544746343304132, 'uuid_mslong': 903894136687250242}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0c8b46ed-a3cd-4b42-922b-1da49ab30bc4'}, 'oper': 'CREATE', 'request-id': 'req-9322483c-9529-4c55-bbf5-d99a46b7c326', 'type': 'analytics_node', 'uuid': '0c8b46ed-a3cd-4b42-922b-1da49ab30bc4'} 09/01/2025 02:52:41.052 7fe3d7a0f198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:52:41.053 7fe3d7a0f198 [contrail-device-manager] [INFO]: Session Event: TCP Connected 09/01/2025 02:52:41.053 7fe3d7a0f198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:52:41.053 7fe3d7a0f198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:41.053 7fe3d7a0f198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3996-2. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.48:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:52:41.053 7fe3d7a0f198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:52:41.053 7fe3d7a0f198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/01/2025 02:52:41.054 7fe3d7a0f198 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 09/01/2025 02:52:41.054 7fe3d7a0f198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/01/2025 02:52:41.054 7fe3d7a0f198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/01/2025 02:52:41.054 7fe3d7a0f198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/01/2025 02:52:41.054 7fe3d7a12438 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/01/2025 02:52:41.055 7fe3d7a12438 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/01/2025 02:52:50.097 7fe3d7a12ac8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001016 09/01/2025 02:52:50.098 7fe3d7a12ac8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/01/2025 02:52:50.098 7fe3d79b7e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 09/01/2025 02:52:50.098 7fe3d79b7e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 09/01/2025 02:52:50.098 7fe3d79b7e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:52:50.098 7fe3d79b7e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 02:52:50.098 7fe3d79b7e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 09/01/2025 02:52:50.099 7fe3d7a12cc0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/01/2025 02:52:50.100 7fe3d7a12cc0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/01/2025 02:52:54.569 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-2.'], 'id_perms': {'created': '2025-09-01T02:52:54.550173', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:52:54.550173', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10075690525769497906, 'uuid_mslong': 15874297488839427795}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dc4cd575-e126-46d3-8bd4-0b265f427d32'}, 'oper': 'CREATE', 'request-id': 'req-eae788f2-b1b7-4d6a-9d13-d29be711593b', 'type': 'analytics_alarm_node', 'uuid': 'dc4cd575-e126-46d3-8bd4-0b265f427d32'} 09/01/2025 02:52:54.598 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-1.'], 'id_perms': {'created': '2025-09-01T02:52:54.583360', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:52:54.583360', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10064605993898106689, 'uuid_mslong': 1917071208486159036}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1a9ace1b-66f2-42bc-8bac-a9d3ae905f41'}, 'oper': 'CREATE', 'request-id': 'req-1e04a02e-4cd0-4dc9-bebc-e7308b3f4d38', 'type': 'analytics_alarm_node', 'uuid': '1a9ace1b-66f2-42bc-8bac-a9d3ae905f41'} 09/01/2025 02:52:54.631 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-3.'], 'id_perms': {'created': '2025-09-01T02:52:54.625800', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:52:54.625800', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12685756876518986754, 'uuid_mslong': 5571148175120551813}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4d50b1bf-8120-4f85-b00c-dcaacbe21402'}, 'oper': 'CREATE', 'request-id': 'req-5abcb46b-bc56-4b8c-a4f7-9981d004a304', 'type': 'analytics_alarm_node', 'uuid': '4d50b1bf-8120-4f85-b00c-dcaacbe21402'} 09/01/2025 02:53:00.048 7fe3d7a12ac8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001296 09/01/2025 02:53:00.048 7fe3d7a12ac8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/01/2025 02:53:00.049 7fe3d7a0ac18 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 09/01/2025 02:53:00.049 7fe3d7a0ac18 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 09/01/2025 02:53:00.049 7fe3d7a0ac18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:53:00.049 7fe3d7a0ac18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 02:53:03.307 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-2.'], 'id_perms': {'created': '2025-09-01T02:53:03.295320', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:53:03.295320', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12275581577176241160, 'uuid_mslong': 14138027032155669049}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c4345cad-e482-4239-aa5b-a067d6eb5408'}, 'oper': 'CREATE', 'request-id': 'req-a8817e8b-1a32-47fa-b823-8ca25d3a3f64', 'type': 'analytics_snmp_node', 'uuid': 'c4345cad-e482-4239-aa5b-a067d6eb5408'} 09/01/2025 02:53:03.329 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-1.'], 'id_perms': {'created': '2025-09-01T02:53:03.298710', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:53:03.298710', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9783303147595751169, 'uuid_mslong': 13045124635484309113}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b50997bd-6cbd-4679-87c5-4660ea67df01'}, 'oper': 'CREATE', 'request-id': 'req-77c8d31a-d710-427d-8a7e-d63bc3dde79f', 'type': 'analytics_snmp_node', 'uuid': 'b50997bd-6cbd-4679-87c5-4660ea67df01'} 09/01/2025 02:53:03.387 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3996-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3996-3.'], 'id_perms': {'created': '2025-09-01T02:53:03.367523', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:53:03.367523', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12249543453717589901, 'uuid_mslong': 599652296990737909}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '08526496-46e1-45f5-a9ff-1edecb8c6f8d'}, 'oper': 'CREATE', 'request-id': 'req-44bb3e9f-5bc2-4c25-bea2-db524a316e57', 'type': 'analytics_snmp_node', 'uuid': '08526496-46e1-45f5-a9ff-1edecb8c6f8d'} 09/01/2025 02:54:40.217 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-fb246a35-697e-4abc-966e-7737c27ce890', 'type': 'global_system_config', 'uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c'} 09/01/2025 02:54:40.292 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-3fc53b04-f05f-4c0d-8276-10ca45dfe48c', 'type': 'global_system_config', 'uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c'} 09/01/2025 02:54:40.377 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3996-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-1'], 'id_perms': {'created': '2025-09-01T02:54:40.365570', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:54:40.365570', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10169214924355861540, 'uuid_mslong': 1930224882274091694}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1ac9894d-bc91-42ae-8d20-4f184ad4fc24', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.25'}, 'oper': 'CREATE', 'request-id': 'req-ec8d76f5-f807-4c52-955d-8d1cbec19105', 'type': 'virtual_router', 'uuid': '1ac9894d-bc91-42ae-8d20-4f184ad4fc24'} 09/01/2025 02:54:40.460 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3996-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-2'], 'id_perms': {'created': '2025-09-01T02:54:40.450113', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:54:40.450113', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9392150248089123717, 'uuid_mslong': 9175537321764211973}}, 'parent_type': 'global-system-config', 'parent_uuid': '9612667c-30a0-4019-b7ab-28f3da3e8b0c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '7f560e94-7f18-4d05-8257-9ee38e72fb85', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-d4ae51ee-30e0-49d4-a9e2-2dd51b946e3b', 'type': 'virtual_router', 'uuid': '7f560e94-7f18-4d05-8257-9ee38e72fb85'} 09/01/2025 02:54:40.735 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:54:40.721579', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:54:40.721579', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11719750772287484169, 'uuid_mslong': 4726930481350788811}}, 'parent_type': 'virtual-router', 'parent_uuid': '1ac9894d-bc91-42ae-8d20-4f184ad4fc24', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '41996e3a-7ae1-46cb-a2a4-eb3bc64e3109', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:41:99:6e:3a:7a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '8b868106-967a-4fcf-a81e-c42a45d33fb2'}]}, 'oper': 'CREATE', 'request-id': 'req-a1bc4c27-0db5-4520-a6c2-aa0f14585c5d', 'type': 'virtual_machine_interface', 'uuid': '41996e3a-7ae1-46cb-a2a4-eb3bc64e3109'} 09/01/2025 02:54:40.786 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-a1bc4c27-0db5-4520-a6c2-aa0f14585c5d', 'type': 'virtual_machine_interface', 'uuid': '41996e3a-7ae1-46cb-a2a4-eb3bc64e3109'} 09/01/2025 02:54:40.842 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-2', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:54:40.828482', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:54:40.828482', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13774093898565881202, 'uuid_mslong': 5449318275197845907}}, 'parent_type': 'virtual-router', 'parent_uuid': '7f560e94-7f18-4d05-8257-9ee38e72fb85', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4b9fde19-7d36-4193-bf27-696c26e50d72', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:4b:9f:de:19:7d']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '8b868106-967a-4fcf-a81e-c42a45d33fb2'}]}, 'oper': 'CREATE', 'request-id': 'req-2a94ea2d-b6f6-4d89-a2b4-8376f10d2246', 'type': 'virtual_machine_interface', 'uuid': '4b9fde19-7d36-4193-bf27-696c26e50d72'} 09/01/2025 02:54:40.892 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3996-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-2a94ea2d-b6f6-4d89-a2b4-8376f10d2246', 'type': 'virtual_machine_interface', 'uuid': '4b9fde19-7d36-4193-bf27-696c26e50d72'} 09/01/2025 02:54:41.290 7fe3d7a16588 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-14db33b9-38e6-4225-8e44-c9f6d2fbf5f7', 'type': 'access_control_list', 'uuid': 'c0ca4e88-2a1f-463c-a4c4-aad232bc07f3'} 09/01/2025 02:55:05.714 7fe3d799af60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:55:35.074 7fe3d79f74e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/01/2025 02:57:07.895 7fe3d7a66588 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map