04/24/2025 00:38:04.703 7f87eca75c08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 04/24/2025 00:38:04.703 7f87eca75c08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 04/24/2025 00:38:04.703 7f87eca75c08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 04/24/2025 00:38:04.703 7f87e41152e8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 04/24/2025 00:38:04.704 7f87e41152e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 04/24/2025 00:38:04.704 7f87e41152e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 04/24/2025 00:38:04.704 7f87e41152e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 04/24/2025 00:38:04.704 7f87e41152e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.705 7f87e41152e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 00:38:04.705 7f87e41152e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.705 7f87e41152e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.705 7f87e41152e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 04/24/2025 00:38:04.705 7f87e41152e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.756 7f87eca75c08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.41:5673//', 'pyamqp://guest:guest@10.0.0.27:5673//', 'pyamqp://guest:guest@10.0.0.23:5673//'] 04/24/2025 00:38:04.765 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/24/2025 00:38:04.765 7f87e4066588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.765 7f87e4066588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:38:04.765 7f87e4066588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.766 7f87e4066588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.766 7f87e4066588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:38:04.766 7f87e4066588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.769 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/24/2025 00:38:04.769 7f87e4066240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.769 7f87e4066240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 00:38:04.769 7f87e4066240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.794 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/24/2025 00:38:04.794 7f87e40660f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.795 7f87e40660f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:38:04.795 7f87e40660f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.798 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/24/2025 00:38:04.798 7f87f2919978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.798 7f87f2919978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 04/24/2025 00:38:04.798 7f87f2919978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.813 7f87eca75c08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 04/24/2025 00:38:04.814 7f87eca75c08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/24/2025 00:38:04.814 7f87eca75c08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/24/2025 00:38:04.814 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/24/2025 00:38:04.815 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 04/24/2025 00:38:04.815 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 04/24/2025 00:38:04.815 7f87e68b5978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 04/24/2025 00:38:04.815 7f87e68b5978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 04/24/2025 00:38:04.816 7f87dfee6588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.816 7f87dfee6588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:04.816 7f87dfee6588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.818 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/24/2025 00:38:04.819 7f87e999e6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 04/24/2025 00:38:04.819 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 04/24/2025 00:38:04.819 7f87e68b5978 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 04/24/2025 00:38:04.857 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/24/2025 00:38:04.865 7f87dfe38630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.866 7f87dfe38630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 04/24/2025 00:38:04.866 7f87dfe38630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.871 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/24/2025 00:38:04.871 7f87dfe3d8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.873 7f87dfe3d8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:04.873 7f87dfe3d8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:04.882 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/24/2025 00:38:04.930 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/24/2025 00:38:08.704 7f87dfdbe588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:08.705 7f87dfdbe588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:08.705 7f87dfdbe588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:08.705 7f87dfdbe588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:08.705 7f87dfdbe588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:08.705 7f87dfdbe588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:08.705 7f87dfdbe588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:08.706 7f87dfdbe588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:11.926 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:38:12.706 7f87dfdef828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:12.707 7f87dfdef828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:12.707 7f87dfdef828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:12.707 7f87dfdef828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:12.708 7f87dfdef828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:12.761 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 04/24/2025 00:38:12.765 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:38:15.066 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:38:16.708 7f87df567978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:16.709 7f87df567978 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:16.709 7f87df567978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:16.709 7f87df567978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:16.709 7f87df567978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:16.709 7f87df567978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:16.709 7f87df567978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:16.710 7f87df567978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:16.785 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:38:16.790 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:38:19.008 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:38:19.932 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 04/24/2025 00:38:19.936 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:38:20.710 7f87dfdd8390 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:20.712 7f87dfdd8390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:20.713 7f87dfdd8390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:20.713 7f87dfdd8390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:20.713 7f87dfdd8390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:20.713 7f87dfdd8390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:20.713 7f87dfdd8390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:22.836 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:38:23.929 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:38:23.937 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:38:24.713 7f87df55ecc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:24.714 7f87df55ecc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:24.714 7f87df55ecc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:24.714 7f87df55ecc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:24.715 7f87df55ecc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:24.716 7f87df55ecc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:24.716 7f87df55ecc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:25.966 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:38:25.968 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 04/24/2025 00:38:25.970 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:38:27.922 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:38:28.714 7f87dfde6cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:28.781 7f87dfde6cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:28.782 7f87dfde6cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:28.782 7f87dfde6cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:38:28.782 7f87dfde6cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:28.782 7f87dfde6cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:28.782 7f87dfde6cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:28.782 7f87dfde6cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.048 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:38:29.056 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:38:29.057 7f87dfdf3eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.057 7f87dfdf3eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:29.057 7f87dfdf3eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.074 7f87eca75c08 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 04/24/2025 00:38:29.075 7f87df58beb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.075 7f87df58beb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 04/24/2025 00:38:29.076 7f87df58beb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.107 7f87df58b780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.107 7f87df58b780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:29.107 7f87df58b780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:29.107 7f87eca75c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 04/24/2025 00:38:29.108 7f87eca75c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 04/24/2025 00:38:29.108 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 04/24/2025 00:38:29.108 7f87eca75c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3195-1. 04/24/2025 00:38:29.111 7f87e999e6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 04/24/2025 00:38:29.171 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3195-1.'] 04/24/2025 00:38:29.185 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 04/24/2025 00:38:29.190 7f87e999e6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 04/24/2025 00:38:29.211 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3195-1.'] 04/24/2025 00:38:29.621 7f87eca75c08 [contrail-device-manager] [DEBUG]: Fetching all containers 04/24/2025 00:38:29.975 7f87eca75c08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 04/24/2025 00:38:30.290 7f87eca75c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3195-1..config_queue 04/24/2025 00:38:30.293 7f87eca75c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3195-1..tftp_queue 04/24/2025 00:38:30.794 7f87eca75c08 [contrail-device-manager] [WARNING]: Process 136 prepared to run in Full mode... 04/24/2025 00:38:30.794 7f87eca75c08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 04/24/2025 00:38:30.794 7f87eca75c08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 04/24/2025 00:38:30.797 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 04/24/2025 00:38:30.797 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 04/24/2025 00:38:30.799 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 04/24/2025 00:38:30.799 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 04/24/2025 00:38:30.800 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 04/24/2025 00:38:30.800 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 04/24/2025 00:38:30.805 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 04/24/2025 00:38:30.805 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 04/24/2025 00:38:30.805 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 04/24/2025 00:38:30.806 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 04/24/2025 00:38:30.806 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 04/24/2025 00:38:30.806 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 04/24/2025 00:38:30.806 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 04/24/2025 00:38:30.807 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 04/24/2025 00:38:30.807 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 04/24/2025 00:38:30.807 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 04/24/2025 00:38:30.807 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 04/24/2025 00:38:30.808 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 04/24/2025 00:38:30.808 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 04/24/2025 00:38:30.813 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 04/24/2025 00:38:30.813 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 04/24/2025 00:38:30.813 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 04/24/2025 00:38:30.813 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 04/24/2025 00:38:30.816 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 04/24/2025 00:38:30.816 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 04/24/2025 00:38:30.816 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 04/24/2025 00:38:30.816 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 04/24/2025 00:38:30.816 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 04/24/2025 00:38:30.817 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 04/24/2025 00:38:30.817 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 04/24/2025 00:38:30.817 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 04/24/2025 00:38:30.817 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 04/24/2025 00:38:30.820 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 04/24/2025 00:38:30.820 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 04/24/2025 00:38:30.821 7f87eca75c08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 04/24/2025 00:38:30.828 7f87eca75c08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 04/24/2025 00:38:30.981 7f87e61db828 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 04/24/2025 00:38:30.982 7f87e61db828 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 04/24/2025 00:38:30.982 7f87decadd68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:30.988 7f87decadd68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/24/2025 00:38:30.988 7f87decadd68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:31.014 7f87dec20978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:31.015 7f87dec20978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/24/2025 00:38:31.015 7f87dec20978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:31.237 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 04/24/2025 00:38:31.243 7f87e999e6d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 04/24/2025 00:38:31.411 7f87e999e6d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3195-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3195-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3195-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 04/24/2025 00:38:32.781 7f87dec19ac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:32.783 7f87dec19ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:32.784 7f87dec19ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 04/24/2025 00:38:32.784 7f87dec19ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:32.784 7f87dec19ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:32.784 7f87dec19ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:32.784 7f87dec19ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:34.173 7f87dec0ccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:34.173 7f87dec0ccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:34.173 7f87dec0ccc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:36.783 7f87deb9dcc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:36.785 7f87deb9dcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:36.786 7f87deb9dcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:36.786 7f87deb9dcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:36.786 7f87deb9dcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:36.786 7f87deb9dcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:36.786 7f87deb9dcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:40.785 7f87deb9d2e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:40.786 7f87deb9d2e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:40.786 7f87deb9d2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:40.786 7f87deb9d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:40.786 7f87deb9d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:40.787 7f87deb9d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:44.787 7f87dec196d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:44.788 7f87dec196d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:44.788 7f87dec196d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:44.788 7f87dec196d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:44.788 7f87dec196d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:38:44.788 7f87dec196d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:44.788 7f87dec196d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:44.789 7f87dec196d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:48.789 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:48.790 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:48.790 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:48.790 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:48.790 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:38:48.790 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:48.790 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:48.791 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:52.791 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:52.792 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:52.792 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:52.792 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:52.792 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:38:52.792 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:52.792 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:52.793 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:56.793 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:38:56.794 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:38:56.794 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:38:56.794 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:56.794 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:38:56.794 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:38:56.794 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:38:56.795 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:00.794 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:00.796 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:00.797 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:00.797 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:00.797 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:00.797 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:00.797 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:04.796 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:04.798 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:04.799 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:04.799 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:04.799 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:04.799 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:04.799 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:08.798 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:08.799 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:08.800 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:08.801 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:08.801 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:08.801 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:08.801 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:12.802 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:12.803 7f87deba0c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:12.803 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:12.803 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:12.803 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:12.803 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:12.803 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:12.804 7f87deba0c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:16.804 7f87deba0eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:16.805 7f87deba0eb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:16.805 7f87deba0eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:16.805 7f87deba0eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:16.806 7f87deba0eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:16.807 7f87deba0eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:20.808 7f87dec80a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:20.809 7f87dec80a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:20.809 7f87dec80a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:20.809 7f87dec80a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:20.810 7f87dec80a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:24.810 7f87dfdd7198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:24.811 7f87dfdd7198 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:24.811 7f87dfdd7198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:24.811 7f87dfdd7198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:24.811 7f87dfdd7198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:24.811 7f87dfdd7198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:24.811 7f87dfdd7198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:24.812 7f87dfdd7198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:28.814 7f87deb37a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:28.818 7f87deb37a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:28.818 7f87deb37a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:28.818 7f87deb37a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:28.819 7f87deb37a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:28.820 7f87deb37a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:28.820 7f87deb37a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:32.818 7f87deb2fac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:32.819 7f87deb2fac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:32.820 7f87deb2fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:32.821 7f87deb2fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:32.821 7f87deb2fac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:32.821 7f87deb2fac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:32.821 7f87deb2fac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:36.820 7f87e9b57c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:36.821 7f87e9b57c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:36.821 7f87e9b57c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:36.821 7f87e9b57c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:36.821 7f87e9b57c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:36.821 7f87e9b57c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:36.821 7f87e9b57c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:36.822 7f87e9b57c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:40.821 7f87deb37438 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:40.822 7f87deb37438 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:40.823 7f87deb37438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:40.824 7f87deb37438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:40.824 7f87deb37438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:44.823 7f87deb37048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:44.824 7f87deb37048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:44.824 7f87deb37048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:44.824 7f87deb37048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:44.824 7f87deb37048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:44.824 7f87deb37048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:44.824 7f87deb37048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:44.825 7f87deb37048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:48.828 7f87deb372e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:48.832 7f87deb372e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:48.832 7f87deb372e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:48.832 7f87deb372e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:48.832 7f87deb372e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:39:48.833 7f87deb372e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:48.833 7f87deb372e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:48.833 7f87deb372e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:48.833 7f87deb372e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:48.833 7f87deb372e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:39:48.834 7f87deb372e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:48.834 7f87deb372e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:48.834 7f87deb372e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:48.834 7f87deb372e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.268 7f87eca75c08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 04/24/2025 00:39:49.269 7f87eca75c08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 04/24/2025 00:39:49.269 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 04/24/2025 00:39:49.269 7f87e9b88240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.269 7f87e9b88240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:39:49.269 7f87e9b88240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.276 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 04/24/2025 00:39:49.386 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 04/24/2025 00:39:49.387 7f87deb8d438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.387 7f87deb8d438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 04/24/2025 00:39:49.387 7f87deb8d438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.394 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 04/24/2025 00:39:49.394 7f87deb8d2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.394 7f87deb8d2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:39:49.394 7f87deb8d2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:49.408 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:50.086 7f87eca75c08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 04/24/2025 00:39:50.090 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:50.098 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:50.987 7f87eca75c08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 04/24/2025 00:39:50.991 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:50.991 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 04/24/2025 00:39:51.046 7f87eca75c08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 04/24/2025 00:39:51.056 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:52.070 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 04/24/2025 00:39:52.076 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:52.083 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:52.833 7f87dea61a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:52.834 7f87dea61a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:52.834 7f87dea61a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:52.834 7f87dea61a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:52.834 7f87dea61a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:39:52.834 7f87dea61a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:52.834 7f87dea61a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:52.835 7f87dea61a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:52.964 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:39:52.967 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:52.972 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:54.095 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 04/24/2025 00:39:54.098 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:54.103 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:55.008 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:39:55.013 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:55.019 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:55.934 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 04/24/2025 00:39:55.938 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:55.943 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:56.834 7f87df58bc18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:39:56.836 7f87df58bc18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:39:56.836 7f87df58bc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:39:56.836 7f87df58bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:56.836 7f87df58bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:39:56.836 7f87df58bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:39:56.836 7f87df58bc18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:39:56.837 7f87df58bc18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:39:56.837 7f87df58bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:56.837 7f87df58bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:39:56.838 7f87df58bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:56.838 7f87df58bc18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:56.838 7f87df58bc18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:39:56.838 7f87df58bc18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:39:58.051 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:39:58.055 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:58.062 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:39:59.008 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 04/24/2025 00:39:59.012 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:39:59.018 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 04/24/2025 00:40:00.837 7f87dea61d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:40:00.838 7f87dea61d68 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:40:00.838 7f87dea61d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:40:00.838 7f87dea61d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:00.838 7f87dea61d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:40:00.838 7f87dea61d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:00.838 7f87dea61d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:40:00.839 7f87dea61d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.918 7f87eca75c08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 04/24/2025 00:40:00.923 7f87eca75c08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 04/24/2025 00:40:00.924 7f87deacc780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.924 7f87deacc780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:40:00.924 7f87deacc780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.969 7f87eca75c08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.41:5673//', 'pyamqp://guest:guest@10.0.0.27:5673//', 'pyamqp://guest:guest@10.0.0.23:5673//'] 04/24/2025 00:40:00.970 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 04/24/2025 00:40:00.970 7f87dea7bac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.971 7f87dea7bac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:40:00.971 7f87dea7bac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.971 7f87dea7bac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.971 7f87dea7bac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:40:00.971 7f87dea7bac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.974 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 04/24/2025 00:40:00.975 7f87dea7b4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.975 7f87dea7b4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:40:00.975 7f87dea7b4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:00.999 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/24/2025 00:40:01.000 7f87dea93048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.000 7f87dea93048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:40:01.000 7f87dea93048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.006 7f87eca75c08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/24/2025 00:40:01.007 7f87dea93390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.007 7f87dea93390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:40:01.007 7f87dea93390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.008 7f87dfdda6d8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 04/24/2025 00:40:01.009 7f87dea9a8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.009 7f87dea9a8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 04/24/2025 00:40:01.009 7f87dea9a8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.013 7f87dfdda6d8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 04/24/2025 00:40:01.013 7f87dea99b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:01.013 7f87dea99b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:40:01.013 7f87dea99b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:04.839 7f87ea471780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 04/24/2025 00:40:04.841 7f87ea471780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:04.842 7f87ea471780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.27:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 04/24/2025 00:40:04.842 7f87ea471780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:04.842 7f87ea471780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:04.842 7f87ea471780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.27:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:40:04.842 7f87ea471780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:08.841 7f87dfdeb240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:40:08.842 7f87dfdeb240 [contrail-device-manager] [INFO]: Session Event: TCP Connected 04/24/2025 00:40:08.842 7f87dfdeb240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:40:08.842 7f87dfdeb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 04/24/2025 00:40:08.843 7f87dfdeb240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 04/24/2025 00:40:08.843 7f87dea94b70 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 04/24/2025 00:40:08.844 7f87dea94b70 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 04/24/2025 00:40:09.557 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.27', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-2.'], 'id_perms': {'created': '2025-04-24T00:40:09.550827', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:09.550827', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10300123662075043220, 'uuid_mslong': 12918503739702329659}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b347beb4-c283-413b-8ef1-63e5f4941d94'}, 'oper': 'CREATE', 'request-id': 'req-6ea14901-a672-4964-ba21-fbe240df16be', 'type': 'analytics_node', 'uuid': 'b347beb4-c283-413b-8ef1-63e5f4941d94'} 04/24/2025 00:40:09.604 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.41', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-1.'], 'id_perms': {'created': '2025-04-24T00:40:09.596434', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:09.596434', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13215715469380259262, 'uuid_mslong': 11815330052295837640}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a3f87c15-0f17-47c8-b767-a733150b8dbe'}, 'oper': 'CREATE', 'request-id': 'req-a066760a-de32-4ebd-a676-c6a17ac286d4', 'type': 'analytics_node', 'uuid': 'a3f87c15-0f17-47c8-b767-a733150b8dbe'} 04/24/2025 00:40:09.620 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-3.'], 'id_perms': {'created': '2025-04-24T00:40:09.596900', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:09.596900', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11991290131536255563, 'uuid_mslong': 6682029495462020993}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5cbb566c-57b2-4781-a669-9ed790dd9a4b'}, 'oper': 'CREATE', 'request-id': 'req-84e3be05-bef0-412a-8472-926002cd6d3a', 'type': 'analytics_node', 'uuid': '5cbb566c-57b2-4781-a669-9ed790dd9a4b'} 04/24/2025 00:40:18.972 7f87deb1c2e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connection Closed 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.41:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:18.973 7f87deb4b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 1 pid = 136 http_port = 8096 start_time = 1745455084704597 collector_name = collector_ip = 10.0.0.41:8086 collector_list = [ 10.0.0.41:8086, 10.0.0.23:8086, 10.0.0.27:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 04/24/2025 00:40:18.974 7f87deb4b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 04/24/2025 00:40:22.977 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 04/24/2025 00:40:22.978 7f87deb4b240 [contrail-device-manager] [INFO]: Session Event: TCP Connected 04/24/2025 00:40:22.979 7f87deb4b240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 04/24/2025 00:40:22.979 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:22.979 7f87deb4b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3195-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.41:5673, 10.0.0.27:5673, 10.0.0.23:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.41:9041, 10.0.0.27:9041, 10.0.0.23:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.41:2181, 10.0.0.27:2181, 10.0.0.23:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.41,10.0.0.27,10.0.0.23:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 04/24/2025 00:40:22.979 7f87deb4b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 04/24/2025 00:40:22.979 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 04/24/2025 00:40:22.979 7f87deb4b240 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 04/24/2025 00:40:22.980 7f87deb4b240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 04/24/2025 00:40:22.980 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 04/24/2025 00:40:22.980 7f87deb4b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 04/24/2025 00:40:22.980 7f87deb58828 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 04/24/2025 00:40:22.981 7f87deb58828 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 04/24/2025 00:40:22.985 7f87deb58630 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 04/24/2025 00:40:22.985 7f87deb58630 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 04/24/2025 00:40:22.986 7f87deb1ccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 04/24/2025 00:40:22.986 7f87deb1ccc0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 04/24/2025 00:40:22.986 7f87deb1ccc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 04/24/2025 00:40:22.986 7f87deb1ccc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 04/24/2025 00:40:22.986 7f87deb1ccc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 04/24/2025 00:40:22.986 7f87dec7e8d0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 04/24/2025 00:40:22.987 7f87dec7e8d0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 04/24/2025 00:40:24.139 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.27', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-2.'], 'id_perms': {'created': '2025-04-24T00:40:24.123968', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:24.123968', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10501890888171682965, 'uuid_mslong': 4977599320299490058}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4513fc35-98ad-4b0a-91be-361f13627c95'}, 'oper': 'CREATE', 'request-id': 'req-7be1b2a8-676a-4e77-ab4d-b1e109d5b4fd', 'type': 'analytics_alarm_node', 'uuid': '4513fc35-98ad-4b0a-91be-361f13627c95'} 04/24/2025 00:40:24.878 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.41', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-1.'], 'id_perms': {'created': '2025-04-24T00:40:24.864744', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:24.864744', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13326800466416387774, 'uuid_mslong': 2027741034156409196}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1c23fbba-2560-416c-b8f2-4e6a714246be'}, 'oper': 'CREATE', 'request-id': 'req-e74a34c5-56ae-4952-81ad-2350731a73d5', 'type': 'analytics_alarm_node', 'uuid': '1c23fbba-2560-416c-b8f2-4e6a714246be'} 04/24/2025 00:40:24.910 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-3.'], 'id_perms': {'created': '2025-04-24T00:40:24.897722', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:24.897722', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13504931577103978460, 'uuid_mslong': 1550487060417498644}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '15846fc4-4106-4e14-bb6b-27b7993d0fdc'}, 'oper': 'CREATE', 'request-id': 'req-e47a6eea-7a21-4697-9976-f09e1d0542b1', 'type': 'analytics_alarm_node', 'uuid': '15846fc4-4106-4e14-bb6b-27b7993d0fdc'} 04/24/2025 00:40:29.871 7f87deb58630 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001296 04/24/2025 00:40:29.872 7f87deb58630 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 04/24/2025 00:40:29.872 7f87dea94e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 04/24/2025 00:40:29.872 7f87dea94e10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 04/24/2025 00:40:29.873 7f87dea94e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 04/24/2025 00:40:29.873 7f87dea94e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 04/24/2025 00:40:34.968 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.27', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-2.'], 'id_perms': {'created': '2025-04-24T00:40:34.959919', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:34.959919', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10724743527476851185, 'uuid_mslong': 12456946860991990128}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'acdff72b-9b58-4970-94d5-f16a866d71f1'}, 'oper': 'CREATE', 'request-id': 'req-20830f95-f41c-4450-92b1-04b6be81c028', 'type': 'analytics_snmp_node', 'uuid': 'acdff72b-9b58-4970-94d5-f16a866d71f1'} 04/24/2025 00:40:35.014 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-3.'], 'id_perms': {'created': '2025-04-24T00:40:35.007637', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:35.007637', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12168604984695191392, 'uuid_mslong': 4149876682677633612}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '399752db-1cad-424c-a8df-91c2f4cdf760'}, 'oper': 'CREATE', 'request-id': 'req-e43751ab-8b0a-40b1-a33f-df957ceb6989', 'type': 'analytics_snmp_node', 'uuid': '399752db-1cad-424c-a8df-91c2f4cdf760'} 04/24/2025 00:40:35.081 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.41', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3195-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3195-1.'], 'id_perms': {'created': '2025-04-24T00:40:35.072608', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:40:35.072608', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10027686901780114529, 'uuid_mslong': 15308860204000887247}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd473ff3e-e796-41cf-8b29-801bb370ec61'}, 'oper': 'CREATE', 'request-id': 'req-26ac3a90-6c6f-43ac-8322-2e81c4eafd2c', 'type': 'analytics_snmp_node', 'uuid': 'd473ff3e-e796-41cf-8b29-801bb370ec61'} 04/24/2025 00:42:14.130 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-69774241-784b-4e50-af06-cab8f66ef662', 'type': 'global_system_config', 'uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69'} 04/24/2025 00:42:14.258 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3195-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-2'], 'id_perms': {'created': '2025-04-24T00:42:14.249907', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:42:14.249907', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11271213443366133945, 'uuid_mslong': 5297775593745042661}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '49857ad5-6ce8-4ce5-9c6b-64e8c77220b9', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-53279d4c-91bc-4ea8-a789-1de686106e53', 'type': 'virtual_router', 'uuid': '49857ad5-6ce8-4ce5-9c6b-64e8c77220b9'} 04/24/2025 00:42:14.590 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-2', 'vhost0'], 'id_perms': {'created': '2025-04-24T00:42:14.574633', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:42:14.574633', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13509088044286373260, 'uuid_mslong': 8122232007070074739}}, 'parent_type': 'virtual-router', 'parent_uuid': '49857ad5-6ce8-4ce5-9c6b-64e8c77220b9', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '70b7f6fa-1cb8-4773-bb79-ec00872a6d8c', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:70:b7:f6:fa:1c']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '4bf22395-048c-42bb-a992-9d2d5c002af1'}]}, 'oper': 'CREATE', 'request-id': 'req-98238685-5072-4343-ab87-1ede97e7ee1f', 'type': 'virtual_machine_interface', 'uuid': '70b7f6fa-1cb8-4773-bb79-ec00872a6d8c'} 04/24/2025 00:42:14.629 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-98238685-5072-4343-ab87-1ede97e7ee1f', 'type': 'virtual_machine_interface', 'uuid': '70b7f6fa-1cb8-4773-bb79-ec00872a6d8c'} 04/24/2025 00:42:14.906 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3195-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-1'], 'id_perms': {'created': '2025-04-24T00:42:14.899502', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:42:14.899502', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13812713714292459607, 'uuid_mslong': 10578911846900190390}}, 'parent_type': 'global-system-config', 'parent_uuid': '53bb4f12-0790-4e5b-bef8-e03651e92b69', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '92cfd846-78a8-4cb6-bfb0-9df0b4918857', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.25'}, 'oper': 'CREATE', 'request-id': 'req-b868f26c-88e3-4d22-8af9-c6806b1400a7', 'type': 'virtual_router', 'uuid': '92cfd846-78a8-4cb6-bfb0-9df0b4918857'} 04/24/2025 00:42:15.142 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-d8c91dac-491d-43dc-93d1-c0f61c60d820', 'type': 'access_control_list', 'uuid': 'da096c43-6680-462e-8f91-a30420f16589'} 04/24/2025 00:42:15.238 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-1', 'vhost0'], 'id_perms': {'created': '2025-04-24T00:42:15.221159', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-04-24T00:42:15.221159', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10310727423908807217, 'uuid_mslong': 7182191834833176069}}, 'parent_type': 'virtual-router', 'parent_uuid': '92cfd846-78a8-4cb6-bfb0-9df0b4918857', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '63ac456b-bff1-4605-8f17-0ff6a5f68231', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:63:ac:45:6b:bf']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '4bf22395-048c-42bb-a992-9d2d5c002af1'}]}, 'oper': 'CREATE', 'request-id': 'req-d7d4dbf0-d4fb-46b7-ae53-3f0085278eb1', 'type': 'virtual_machine_interface', 'uuid': '63ac456b-bff1-4605-8f17-0ff6a5f68231'} 04/24/2025 00:42:15.286 7f87dea93a20 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3195-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-d7d4dbf0-d4fb-46b7-ae53-3f0085278eb1', 'type': 'virtual_machine_interface', 'uuid': '63ac456b-bff1-4605-8f17-0ff6a5f68231'} 04/24/2025 00:42:55.760 7f87dea7ba20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 04/24/2025 00:43:11.901 7f87deac98d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 04/24/2025 01:33:53.969 7f87dea99630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map