12/04/2025 05:37:20.647 7f760b891c48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 12/04/2025 05:37:20.647 7f760b891c48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 12/04/2025 05:37:20.647 7f760b891c48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 12/04/2025 05:37:20.647 7f7602f74048 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1764826640648324 collector_name = collector_ip = collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 12/04/2025 05:37:20.648 7f7602f74048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.695 7f760b891c48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.33:5673//', 'pyamqp://guest:guest@10.0.0.249:5673//'] 12/04/2025 05:37:20.703 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 12/04/2025 05:37:20.703 7f7602ec32e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.704 7f7602ec32e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:37:20.704 7f7602ec32e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.704 7f7602ec32e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.704 7f7602ec32e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:37:20.704 7f7602ec32e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.707 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 12/04/2025 05:37:20.708 7f7602ec3828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.708 7f7602ec3828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/04/2025 05:37:20.708 7f7602ec3828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.732 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 12/04/2025 05:37:20.733 7f7602ed24e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.733 7f7602ed24e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:37:20.733 7f7602ed24e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.738 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 12/04/2025 05:37:20.738 7f7602ed2828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.738 7f7602ed2828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 12/04/2025 05:37:20.738 7f7602ed2828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.763 7f760b891c48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 12/04/2025 05:37:20.763 7f760b891c48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 12/04/2025 05:37:20.763 7f760b891c48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 12/04/2025 05:37:20.763 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 12/04/2025 05:37:20.764 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 12/04/2025 05:37:20.764 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 12/04/2025 05:37:20.764 7f7602ec3ac8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 12/04/2025 05:37:20.764 7f7602ec3ac8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 12/04/2025 05:37:20.764 7f7602d63048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.765 7f7602d63048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:20.765 7f7602d63048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.766 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 12/04/2025 05:37:20.769 7f7602ec3ac8 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 12/04/2025 05:37:20.770 7f76087fa6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 12/04/2025 05:37:20.770 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 12/04/2025 05:37:20.803 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 12/04/2025 05:37:20.805 7f7602cb6f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.805 7f7602cb6f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 12/04/2025 05:37:20.806 7f7602cb6f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.813 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 12/04/2025 05:37:20.814 7f7602cb6f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:20.814 7f7602cb6f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:20.814 7f7602cb6f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:21.822 7f760b891c48 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 12/04/2025 05:37:21.828 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 12/04/2025 05:37:21.865 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 12/04/2025 05:37:22.430 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:37:22.906 7f760b891c48 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 12/04/2025 05:37:22.912 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:37:24.502 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:37:24.649 7f7602c6af60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:24.650 7f7602c6af60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:24.650 7f7602c6af60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:24.651 7f7602c6af60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:25.604 7f760b891c48 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:37:25.611 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:37:27.637 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:37:28.651 7f7602c2d6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:28.653 7f7602c2d6d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:28.653 7f7602c2d6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:28.653 7f7602c2d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:28.653 7f7602c2d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:28.653 7f7602c2d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:28.653 7f7602c2d6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:28.654 7f7602c2d6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:28.998 7f760b891c48 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 12/04/2025 05:37:29.004 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:37:30.429 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:37:31.555 7f760b891c48 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:37:31.560 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:37:32.653 7f7602cb62e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:32.654 7f7602cb62e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:32.654 7f7602cb62e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:32.654 7f7602cb62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:32.655 7f7602cb62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:34.062 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:37:34.064 7f760b891c48 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 12/04/2025 05:37:34.067 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:37:35.738 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:37:36.655 7f7608551048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:36.656 7f7608551048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:36.656 7f7608551048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:36.656 7f7608551048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:36.656 7f7608551048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:36.657 7f7608551048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.049 7f760b891c48 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:37:37.054 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:37:37.055 7f7602cb62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.055 7f7602cb62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:37.055 7f7602cb62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.086 7f760b891c48 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 12/04/2025 05:37:37.088 7f76023da4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.088 7f76023da4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 12/04/2025 05:37:37.088 7f76023da4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.100 7f76023e1390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.100 7f76023e1390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:37.100 7f76023e1390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:37.100 7f760b891c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 12/04/2025 05:37:37.100 7f760b891c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 12/04/2025 05:37:37.100 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 12/04/2025 05:37:37.101 7f760b891c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4841-3. 12/04/2025 05:37:37.104 7f76087fa6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 12/04/2025 05:37:37.153 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4841-3.'] 12/04/2025 05:37:37.178 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 12/04/2025 05:37:37.184 7f76087fa6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 12/04/2025 05:37:37.221 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4841-3.'] 12/04/2025 05:37:37.615 7f760b891c48 [contrail-device-manager] [DEBUG]: Fetching all containers 12/04/2025 05:37:37.683 7f760b891c48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 12/04/2025 05:37:37.836 7f760b891c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4841-3..config_queue 12/04/2025 05:37:37.837 7f760b891c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4841-3..tftp_queue 12/04/2025 05:37:38.239 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 12/04/2025 05:37:38.249 7f76087fa6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 12/04/2025 05:37:38.319 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4841-3.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4841-3..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4841-3..tftp_queue'] 12/04/2025 05:37:38.337 7f760b891c48 [contrail-device-manager] [WARNING]: Process 140 prepared to run in Full mode... 12/04/2025 05:37:38.338 7f760b891c48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 12/04/2025 05:37:38.338 7f760b891c48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 12/04/2025 05:37:38.340 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 12/04/2025 05:37:38.340 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 12/04/2025 05:37:38.341 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 12/04/2025 05:37:38.341 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 12/04/2025 05:37:38.342 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 12/04/2025 05:37:38.342 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 12/04/2025 05:37:38.347 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 12/04/2025 05:37:38.347 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 12/04/2025 05:37:38.347 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 12/04/2025 05:37:38.347 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 12/04/2025 05:37:38.347 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 12/04/2025 05:37:38.347 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 12/04/2025 05:37:38.348 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 12/04/2025 05:37:38.349 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 12/04/2025 05:37:38.350 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 12/04/2025 05:37:38.350 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 12/04/2025 05:37:38.350 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 12/04/2025 05:37:38.350 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 12/04/2025 05:37:38.350 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 12/04/2025 05:37:38.350 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 12/04/2025 05:37:38.351 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 12/04/2025 05:37:38.351 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 12/04/2025 05:37:38.351 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 12/04/2025 05:37:38.351 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 12/04/2025 05:37:38.351 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 12/04/2025 05:37:38.351 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 12/04/2025 05:37:38.352 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 12/04/2025 05:37:38.353 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 12/04/2025 05:37:38.353 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 12/04/2025 05:37:38.353 7f760b891c48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 12/04/2025 05:37:38.355 7f760b891c48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 12/04/2025 05:37:38.370 7f76023176d8 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 12/04/2025 05:37:38.370 7f76023176d8 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 12/04/2025 05:37:38.424 7f7601b5dd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:38.424 7f7601b5dd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 12/04/2025 05:37:38.424 7f7601b5dd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:38.436 7f7602c2d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:38.437 7f7602c2d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 12/04/2025 05:37:38.437 7f7602c2d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:39.341 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 12/04/2025 05:37:39.345 7f76087fa6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 12/04/2025 05:37:39.441 7f76087fa6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-4841-3.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4841-3..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-4841-3..tftp_queue', 'device_manager_ztp.ztp_queue'] 12/04/2025 05:37:40.657 7f7601a444e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:40.658 7f7601a444e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:40.659 7f7601a444e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:40.659 7f7601a444e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 12/04/2025 05:37:40.659 7f7601a444e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:40.659 7f7601a444e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:40.659 7f7601a444e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:40.659 7f7601a444e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:44.567 7f76019ec588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:44.567 7f76019ec588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:44.568 7f76019ec588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:44.659 7f760196aac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:44.660 7f760196aac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:44.660 7f760196aac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:44.660 7f760196aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:44.660 7f760196aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:37:44.660 7f760196aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:44.661 7f760196aac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:44.661 7f760196aac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:44.662 7f760196aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:44.662 7f760196aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:44.662 7f760196aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:44.662 7f760196aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:44.662 7f760196aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:44.662 7f760196aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:48.663 7f760196bc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:48.665 7f760196bc18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:48.666 7f760196bc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:48.666 7f760196bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:48.666 7f760196bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:37:48.666 7f760196bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:48.666 7f760196bc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:48.667 7f760196bc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:48.667 7f760196bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:48.667 7f760196bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:48.667 7f760196bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:48.667 7f760196bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:48.667 7f760196bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:48.668 7f760196bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:52.667 7f7602ed20f0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:52.669 7f7602ed20f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:52.669 7f7602ed20f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:52.669 7f7602ed20f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:52.669 7f7602ed20f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:37:52.669 7f7602ed20f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:52.669 7f7602ed20f0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:52.670 7f7602ed20f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:56.669 7f760196b630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:37:56.671 7f760196b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:56.672 7f760196b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:37:56.672 7f760196b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:56.672 7f760196b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:37:56.672 7f760196b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:37:56.672 7f760196b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:00.671 7f760196a780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:00.672 7f760196a780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:00.673 7f760196a780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:00.674 7f760196a780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:00.674 7f760196a780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:00.674 7f760196a780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:00.674 7f760196a780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:00.674 7f760196a780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:04.673 7f76089c0240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:04.674 7f76089c0240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:04.674 7f76089c0240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:04.674 7f76089c0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:04.674 7f76089c0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:04.674 7f76089c0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:04.674 7f76089c0240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:04.675 7f76089c0240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:08.675 7f760190d2e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:08.676 7f760190d2e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:08.677 7f760190d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:08.678 7f760190d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:08.678 7f760190d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:08.678 7f760190d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:08.678 7f760190d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:12.677 7f7601a080f0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:12.679 7f7601a080f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:12.680 7f7601a080f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:12.680 7f7601a080f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:12.680 7f7601a080f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:12.680 7f7601a080f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:12.680 7f7601a080f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:12.680 7f7601a080f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:16.679 7f7603b12198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:16.680 7f7603b12198 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:16.681 7f7603b12198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:16.682 7f7603b12198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:16.682 7f7603b12198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:16.682 7f7603b12198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:20.681 7f760896a240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:20.682 7f760896a240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:20.683 7f760896a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:20.683 7f760896a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:20.683 7f760896a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:20.683 7f760896a240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:20.683 7f760896a240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:20.683 7f760896a240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:24.683 7f760196ae10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:24.684 7f760196ae10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:24.685 7f760196ae10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:24.685 7f760196ae10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:24.685 7f760196ae10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:24.685 7f760196ae10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:24.685 7f760196ae10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:24.685 7f760196ae10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:28.684 7f76019be828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:28.685 7f76019be828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:28.686 7f76019be828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:28.686 7f76019be828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:28.686 7f76019be828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:28.686 7f76019be828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:28.686 7f76019be828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:28.686 7f76019be828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.481 7f760b891c48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 12/04/2025 05:38:29.481 7f760b891c48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 12/04/2025 05:38:29.481 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 12/04/2025 05:38:29.481 7f76089b2c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.482 7f76089b2c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:38:29.482 7f76089b2c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.483 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 12/04/2025 05:38:29.522 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 12/04/2025 05:38:29.523 7f7601a93eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.523 7f7601a93eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 12/04/2025 05:38:29.523 7f7601a93eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.528 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 12/04/2025 05:38:29.528 7f7601a93d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.529 7f7601a93d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:38:29.529 7f7601a93d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:29.545 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:30.620 7f760b891c48 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 12/04/2025 05:38:30.624 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:30.630 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:32.053 7f760b891c48 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 12/04/2025 05:38:32.058 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:32.058 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 12/04/2025 05:38:32.145 7f760b891c48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 12/04/2025 05:38:32.151 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:32.686 7f760189cb70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:32.687 7f760189cb70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:32.687 7f760189cb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:32.687 7f760189cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:32.687 7f760189cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:38:32.687 7f760189cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:32.688 7f760189cb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:32.695 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 12/04/2025 05:38:32.704 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:32.715 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:33.611 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:38:33.616 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:33.623 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:34.698 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 12/04/2025 05:38:34.702 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:34.715 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:35.632 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:38:35.636 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:35.647 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:36.620 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 12/04/2025 05:38:36.628 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:36.635 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:36.687 7f760184da20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:36.689 7f760184da20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:36.690 7f760184da20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:36.690 7f760184da20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:36.690 7f760184da20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:38:36.690 7f760184da20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:36.690 7f760184da20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:36.691 7f760184da20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:36.692 7f760184da20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:36.692 7f760184da20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 12/04/2025 05:38:36.692 7f760184da20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:36.692 7f760184da20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:36.692 7f760184da20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:36.692 7f760184da20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:37.568 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:38:37.573 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:37.584 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:38.749 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 12/04/2025 05:38:38.761 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:38.809 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 12/04/2025 05:38:39.943 7f760b891c48 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 12/04/2025 05:38:39.947 7f760b891c48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 12/04/2025 05:38:39.948 7f7603b12ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:39.948 7f7603b12ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:39.948 7f7603b12ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:39.996 7f760b891c48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.33:5673//', 'pyamqp://guest:guest@10.0.0.249:5673//'] 12/04/2025 05:38:39.997 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 12/04/2025 05:38:39.997 7f760185fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:39.997 7f760185fa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:38:39.997 7f760185fa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:39.997 7f760185fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:39.998 7f760185fa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:38:39.998 7f760185fa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.002 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 12/04/2025 05:38:40.002 7f760185fcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.002 7f760185fcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:40.002 7f760185fcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.048 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 12/04/2025 05:38:40.049 7f760185e198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.049 7f760185e198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:38:40.049 7f760185e198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.054 7f760b891c48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 12/04/2025 05:38:40.054 7f760185e438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.055 7f760185e438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:40.055 7f760185e438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.056 7f760185ecc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 12/04/2025 05:38:40.056 7f76018a0a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.056 7f76018a0a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 12/04/2025 05:38:40.056 7f76018a0a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.061 7f760185ecc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 12/04/2025 05:38:40.062 7f7601851c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.062 7f7601851c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:40.062 7f7601851c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.660 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-bc2e9f16-7376-4c1b-b2aa-e475953310dc', 'type': 'global_system_config', 'uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90'} 12/04/2025 05:38:40.690 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:40.691 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.692 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:40.692 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.692 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:40.692 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:40.692 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:44.691 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:44.692 7f76018a1c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:44.693 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:44.694 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:44.694 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:48.694 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:48.695 7f76018a1c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:48.695 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:48.695 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:48.696 7f76018a1c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:48.697 7f76018a1c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:52.696 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:52.697 7f7602c5f588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:52.697 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:52.697 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:52.697 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:52.697 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:52.697 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:52.698 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:56.697 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:38:56.698 7f7602c5f588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:38:56.699 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:56.700 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:38:56.700 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:38:56.700 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:00.699 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:00.700 7f7602c5f588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:00.700 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:00.700 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:00.700 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:00.700 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:00.700 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:00.701 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:04.701 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:04.702 7f7602c5f588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:04.702 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:04.702 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:04.702 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:04.702 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:04.702 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:04.703 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:08.702 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:08.704 7f7602c5f588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:08.704 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:08.704 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:08.706 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:08.706 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:08.706 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:08.706 7f7602c5f588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:08.706 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:08.707 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:08.707 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:08.707 7f7602c5f588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:08.707 7f7602c5f588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:08.707 7f7602c5f588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:10.989 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'obj_dict': {'database_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'id_perms': {'created': '2025-12-04T05:39:10.982399', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:10.982399', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11840429031001062458, 'uuid_mslong': 15116717040683139979}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd1c95e0e-8403-438b-a451-a7786d0a7c3a'}, 'oper': 'CREATE', 'request-id': 'req-9532683c-b975-413b-9e9e-181e4cd5937d', 'type': 'database_node', 'uuid': 'd1c95e0e-8403-438b-a451-a7786d0a7c3a'} 12/04/2025 05:39:11.549 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'obj_dict': {'database_node_ip_address': '10.0.0.33', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'id_perms': {'created': '2025-12-04T05:39:11.540932', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:11.540932', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12398479746882680153, 'uuid_mslong': 12928015370508783510}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b369897b-e1f7-4f96-ac10-3fa3cbd13559'}, 'oper': 'CREATE', 'request-id': 'req-2341c24e-e021-4b1c-86d2-2306754b3478', 'type': 'database_node', 'uuid': 'b369897b-e1f7-4f96-ac10-3fa3cbd13559'} 12/04/2025 05:39:11.680 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'id_perms': {'created': '2025-12-04T05:39:11.658129', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:11.658129', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11924997430976435185, 'uuid_mslong': 7405536362181641859}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '66c5c015-cf13-4683-a57e-19fb6ccbd3f1'}, 'oper': 'CREATE', 'request-id': 'req-33db289a-0cc7-4360-b555-5c3dd1314950', 'type': 'database_node', 'uuid': '66c5c015-cf13-4683-a57e-19fb6ccbd3f1'} 12/04/2025 05:39:12.707 7f76018a4cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:12.708 7f76018a4cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:12.708 7f76018a4cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:12.709 7f76018a4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:12.710 7f76018a4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:12.710 7f76018a4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:16.709 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:16.710 7f760979b630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:16.710 7f760979b630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:16.710 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:16.710 7f760979b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:16.710 7f760979b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:16.710 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:16.711 7f760979b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:20.710 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:20.712 7f760979b630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:20.712 7f760979b630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:20.712 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:20.712 7f760979b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:20.712 7f760979b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:20.712 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:20.713 7f760979b630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:24.713 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:24.714 7f7602c5fac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:24.714 7f7602c5fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:24.715 7f7602c5fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:24.716 7f7602c5fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:28.716 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:28.716 7f7602c5fac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:28.717 7f7602c5fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:28.718 7f7602c5fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:28.718 7f7602c5fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:32.717 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:32.718 7f7601860b70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:32.718 7f7601860b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:32.718 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:32.718 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:32.719 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:36.719 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:36.720 7f7601860b70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 12/04/2025 05:39:36.720 7f7601860b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:36.721 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:36.721 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:36.721 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:36.721 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 12/04/2025 05:39:36.721 7f7601860b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 12/04/2025 05:39:36.721 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:36.722 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.33:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 12/04/2025 05:39:36.722 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:36.722 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:36.722 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.33:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:36.722 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:40.721 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [INFO]: Session Event: TCP Connected 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/04/2025 05:39:40.722 7f7601860b70 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 12/04/2025 05:39:40.723 7f7601860b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/04/2025 05:39:40.723 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/04/2025 05:39:40.723 7f7601860b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/04/2025 05:39:40.723 7f760185f048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/04/2025 05:39:40.724 7f760185f048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/04/2025 05:39:41.861 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'id_perms': {'created': '2025-12-04T05:39:41.852301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:41.852301', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13174081360697793463, 'uuid_mslong': 579224449466189744}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0809d191-63a7-47b0-b6d3-bd32c27c9bb7'}, 'oper': 'CREATE', 'request-id': 'req-0fd24ac9-0a3b-4fc5-bac8-2297992d3ac1', 'type': 'analytics_node', 'uuid': '0809d191-63a7-47b0-b6d3-bd32c27c9bb7'} 12/04/2025 05:39:41.952 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.33', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'id_perms': {'created': '2025-12-04T05:39:41.933389', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:41.933389', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12247638337486041548, 'uuid_mslong': 801300045874085897}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0b1eca25-70d4-4009-a9f8-5a2d5b4fb1cc'}, 'oper': 'CREATE', 'request-id': 'req-f692cc61-1410-46dd-8225-8333f45d1740', 'type': 'analytics_node', 'uuid': '0b1eca25-70d4-4009-a9f8-5a2d5b4fb1cc'} 12/04/2025 05:39:41.963 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'id_perms': {'created': '2025-12-04T05:39:41.938742', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:41.938742', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11603757275769920164, 'uuid_mslong': 12754508066127627509}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b1011d82-8664-44f5-a108-d3c3521edaa4'}, 'oper': 'CREATE', 'request-id': 'req-bb7ab2d7-7bc7-4c56-910f-bb4ebf531432', 'type': 'analytics_node', 'uuid': 'b1011d82-8664-44f5-a108-d3c3521edaa4'} 12/04/2025 05:39:54.344 7f76018552e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connection Closed 12/04/2025 05:39:54.344 7f7601863a20 [contrail-device-manager] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 12/04/2025 05:39:54.345 7f7601863a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 12/04/2025 05:39:54.346 7f7601863a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:54.346 7f7601863a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.249:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 12/04/2025 05:39:54.346 7f7601863a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:54.346 7f7601863a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:54.346 7f7601863a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 1 pid = 140 http_port = 8096 start_time = 1764826640648324 collector_name = collector_ip = 10.0.0.249:8086 collector_list = [ 10.0.0.33:8086, 10.0.0.249:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 12/04/2025 05:39:54.346 7f7601863a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 12/04/2025 05:39:57.854 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'id_perms': {'created': '2025-12-04T05:39:57.843385', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:57.843385', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12732304322359427517, 'uuid_mslong': 1323705367154409900}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '125ebf04-a484-41ac-b0b2-3b52926781bd'}, 'oper': 'CREATE', 'request-id': 'req-a59c7d01-6734-4281-aac5-6f82d51abd96', 'type': 'analytics_alarm_node', 'uuid': '125ebf04-a484-41ac-b0b2-3b52926781bd'} 12/04/2025 05:39:58.346 7f760185fe10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [INFO]: Session Event: TCP Connected 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4841-3. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.33:5673, 10.0.0.249:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.33:9041, 10.0.0.249:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.33:2181, 10.0.0.249:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.38,10.0.0.33,10.0.0.249:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 12/04/2025 05:39:58.348 7f760185fe10 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 12/04/2025 05:39:58.349 7f760185fe10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 12/04/2025 05:39:58.349 7f760185fe10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/04/2025 05:39:58.349 7f760185fe10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 12/04/2025 05:39:58.349 7f7601774240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/04/2025 05:39:58.350 7f7601774240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/04/2025 05:39:58.357 7f76017746d8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001016 12/04/2025 05:39:58.358 7f76017746d8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/04/2025 05:39:58.358 7f76018a1c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 12/04/2025 05:39:58.358 7f76018a1c18 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 12/04/2025 05:39:58.358 7f76018a1c18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 12/04/2025 05:39:58.358 7f76018a1c18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/04/2025 05:39:58.359 7f76018a1c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 12/04/2025 05:39:58.360 7f7608822e10 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 12/04/2025 05:39:58.361 7f7608822e10 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 12/04/2025 05:39:59.298 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.33', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'id_perms': {'created': '2025-12-04T05:39:59.261302', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:59.261302', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12670308339255723922, 'uuid_mslong': 15903821989638327670}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dcb5b9d6-6770-4d76-afd5-fa4dec01c792'}, 'oper': 'CREATE', 'request-id': 'req-102e0f12-7626-457f-97e5-b69d6e72b4fc', 'type': 'analytics_alarm_node', 'uuid': 'dcb5b9d6-6770-4d76-afd5-fa4dec01c792'} 12/04/2025 05:39:59.315 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'id_perms': {'created': '2025-12-04T05:39:59.306073', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:39:59.306073', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11488209599260457081, 'uuid_mslong': 10058638612648970619}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8b977686-76f8-497b-9f6e-51c36d26f879'}, 'oper': 'CREATE', 'request-id': 'req-1c42f026-698b-48dd-a301-a9d1050b6758', 'type': 'analytics_alarm_node', 'uuid': '8b977686-76f8-497b-9f6e-51c36d26f879'} 12/04/2025 05:40:05.002 7f76017746d8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001296 12/04/2025 05:40:05.003 7f76017746d8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 12/04/2025 05:40:05.003 7f76018fb438 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 12/04/2025 05:40:05.003 7f76018fb438 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 12/04/2025 05:40:05.003 7f76018fb438 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 12/04/2025 05:40:05.004 7f76018fb438 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 12/04/2025 05:40:11.840 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-1.'], 'id_perms': {'created': '2025-12-04T05:40:11.822158', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:40:11.822158', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12682118402031664156, 'uuid_mslong': 1475408480706249519}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1479b432-3718-432f-afff-ef7e708abc1c'}, 'oper': 'CREATE', 'request-id': 'req-572188e0-5d74-4762-9169-98360e2e32f3', 'type': 'analytics_snmp_node', 'uuid': '1479b432-3718-432f-afff-ef7e708abc1c'} 12/04/2025 05:40:11.841 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.249', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-3.'], 'id_perms': {'created': '2025-12-04T05:40:11.819712', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:40:11.819712', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13647041900299845806, 'uuid_mslong': 9167107388262731176}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '7f381b99-a742-4da8-bd64-084d9e3e80ae'}, 'oper': 'CREATE', 'request-id': 'req-be5029a1-bf84-47e9-b835-e91f61d2a1bc', 'type': 'analytics_snmp_node', 'uuid': '7f381b99-a742-4da8-bd64-084d9e3e80ae'} 12/04/2025 05:40:11.947 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.33', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4841-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4841-2.'], 'id_perms': {'created': '2025-12-04T05:40:11.929265', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:40:11.929265', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9939310799204890736, 'uuid_mslong': 6161143387999784258}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5580c744-abf9-4d42-89ef-8682dacfb870'}, 'oper': 'CREATE', 'request-id': 'req-82ac10f5-4479-48bb-9c20-c1b3a14fe2ae', 'type': 'analytics_snmp_node', 'uuid': '5580c744-abf9-4d42-89ef-8682dacfb870'} 12/04/2025 05:42:01.481 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-be5ec72d-1682-4308-9c01-204f28589deb', 'type': 'global_system_config', 'uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90'} 12/04/2025 05:42:01.620 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-4841-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-1'], 'id_perms': {'created': '2025-12-04T05:42:01.614465', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:42:01.614465', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10732399027115777904, 'uuid_mslong': 6596989920659655528}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b8d3761-67c0-4b68-94f1-240d6efbdf70', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-d85c8f82-e068-485c-954f-47d6e5b0e6f9', 'type': 'virtual_router', 'uuid': '5b8d3761-67c0-4b68-94f1-240d6efbdf70'} 12/04/2025 05:42:01.979 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-1', 'vhost0'], 'id_perms': {'created': '2025-12-04T05:42:01.948176', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:42:01.948176', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10680106441282484398, 'uuid_mslong': 16195354831759951035}}, 'parent_type': 'virtual-router', 'parent_uuid': '5b8d3761-67c0-4b68-94f1-240d6efbdf70', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e0c17569-ae75-44bb-9437-5c390400f8ae', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:e0:c1:75:69:ae']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '21a3c4c2-12ed-47bc-b26e-d444ce6f3529'}]}, 'oper': 'CREATE', 'request-id': 'req-4b9267f7-b997-4f44-8114-436e05a4f31e', 'type': 'virtual_machine_interface', 'uuid': 'e0c17569-ae75-44bb-9437-5c390400f8ae'} 12/04/2025 05:42:02.017 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-4b9267f7-b997-4f44-8114-436e05a4f31e', 'type': 'virtual_machine_interface', 'uuid': 'e0c17569-ae75-44bb-9437-5c390400f8ae'} 12/04/2025 05:42:02.510 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-4841-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-2'], 'id_perms': {'created': '2025-12-04T05:42:02.500527', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:42:02.500527', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10098547509763905766, 'uuid_mslong': 3598570679185522896}}, 'parent_type': 'global-system-config', 'parent_uuid': '19b098a2-dd1f-44bc-bfd8-4a72d7b91e90', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '31f0b0f7-77fd-40d0-8c25-3f74aae218e6', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.129'}, 'oper': 'CREATE', 'request-id': 'req-b898db02-507f-429f-8c1c-a1afd96cec6d', 'type': 'virtual_router', 'uuid': '31f0b0f7-77fd-40d0-8c25-3f74aae218e6'} 12/04/2025 05:42:02.537 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-10d4d6ba-d369-4c1c-84e2-ab0d618b19e5', 'type': 'access_control_list', 'uuid': '1ff0e829-8193-4e2c-83a8-1e365d11920d'} 12/04/2025 05:42:02.879 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-2', 'vhost0'], 'id_perms': {'created': '2025-12-04T05:42:02.858332', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-12-04T05:42:02.858332', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11047913691058399443, 'uuid_mslong': 16957776125690204088}}, 'parent_type': 'virtual-router', 'parent_uuid': '31f0b0f7-77fd-40d0-8c25-3f74aae218e6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'eb561f8a-485d-4fb8-9952-12f7b8b5e8d3', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:eb:56:1f:8a:48']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '21a3c4c2-12ed-47bc-b26e-d444ce6f3529'}]}, 'oper': 'CREATE', 'request-id': 'req-1f3328df-43bd-4531-8923-952715713682', 'type': 'virtual_machine_interface', 'uuid': 'eb561f8a-485d-4fb8-9952-12f7b8b5e8d3'} 12/04/2025 05:42:02.934 7f760185e8d0 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4841-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-1f3328df-43bd-4531-8923-952715713682', 'type': 'virtual_machine_interface', 'uuid': 'eb561f8a-485d-4fb8-9952-12f7b8b5e8d3'}