08/04/2025 05:33:09.601 7f4519439d48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 08/04/2025 05:33:09.601 7f4519439d48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 08/04/2025 05:33:09.601 7f4519439d48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 08/04/2025 05:33:09.602 7f4510af92e8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 08/04/2025 05:33:09.602 7f4510af92e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 08/04/2025 05:33:09.602 7f4510af92e8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 08/04/2025 05:33:09.602 7f4510af92e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 08/04/2025 05:33:09.602 7f4510af92e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.602 7f4510af92e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/04/2025 05:33:09.603 7f4510af92e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.603 7f4510af92e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.603 7f4510af92e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1754285589602575 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 08/04/2025 05:33:09.603 7f4510af92e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.650 7f4519439d48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 08/04/2025 05:33:09.658 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/04/2025 05:33:09.659 7f4510a4b588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.659 7f4510a4b588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:33:09.659 7f4510a4b588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.659 7f4510a4b588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.660 7f4510a4b588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:33:09.660 7f4510a4b588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.663 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/04/2025 05:33:09.663 7f4510a4b240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.663 7f4510a4b240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/04/2025 05:33:09.663 7f4510a4b240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.755 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/04/2025 05:33:09.756 7f4510a4b0f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.756 7f4510a4b0f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:33:09.756 7f4510a4b0f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.759 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/04/2025 05:33:09.759 7f451f2fd978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.759 7f451f2fd978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/04/2025 05:33:09.759 7f451f2fd978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.788 7f4519439d48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 08/04/2025 05:33:09.789 7f4519439d48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 08/04/2025 05:33:09.789 7f4519439d48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 08/04/2025 05:33:09.789 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/04/2025 05:33:09.790 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 08/04/2025 05:33:09.790 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/04/2025 05:33:09.790 7f4513299978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 08/04/2025 05:33:09.790 7f4513299978 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 08/04/2025 05:33:09.790 7f45108e8588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.791 7f45108e8588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:09.791 7f45108e8588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.793 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/04/2025 05:33:09.796 7f4513299978 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 08/04/2025 05:33:09.796 7f45163826d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/04/2025 05:33:09.796 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 08/04/2025 05:33:09.862 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/04/2025 05:33:09.863 7f45108b5438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.863 7f45108b5438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 08/04/2025 05:33:09.863 7f45108b5438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.870 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/04/2025 05:33:09.870 7f45108b4e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:09.870 7f45108b4e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:09.870 7f45108b4e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:10.889 7f4519439d48 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 08/04/2025 05:33:10.897 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/04/2025 05:33:10.929 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/04/2025 05:33:11.085 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:33:12.737 7f4519439d48 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 08/04/2025 05:33:12.746 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:33:13.603 7f45107cb390 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:13.604 7f45107cb390 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:13.604 7f45107cb390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:13.605 7f45107cb390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:13.606 7f45107cb390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:15.813 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:33:16.745 7f4519439d48 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:33:16.751 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:33:17.606 7f45107b5cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:17.608 7f45107b5cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:17.609 7f45107b5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:17.609 7f45107b5cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:17.609 7f45107b5cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:17.609 7f45107b5cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:17.609 7f45107b5cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:18.994 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:33:19.753 7f4519439d48 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 08/04/2025 05:33:19.758 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:33:21.609 7f45107d8828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:21.610 7f45107d8828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:21.611 7f45107d8828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:21.611 7f45107d8828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:21.611 7f45107d8828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:21.611 7f45107d8828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:21.611 7f45107d8828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:21.611 7f45107d8828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:21.816 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:33:22.773 7f4519439d48 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:33:22.778 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:33:24.688 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:33:24.689 7f4519439d48 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 08/04/2025 05:33:24.693 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:33:25.610 7f4510763c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:25.612 7f4510763c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:25.613 7f4510763c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:33:25.613 7f4510763c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:25.613 7f4510763c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:25.613 7f4510763c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:25.613 7f4510763c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:26.860 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:33:28.750 7f4519439d48 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:33:28.754 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:33:28.754 7f45116abcc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:28.755 7f45116abcc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:28.755 7f45116abcc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:28.777 7f4519439d48 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 08/04/2025 05:33:28.777 7f4510768240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:28.778 7f4510768240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 08/04/2025 05:33:28.778 7f4510768240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:28.791 7f45107700f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:28.791 7f45107700f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:28.792 7f45107700f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:28.792 7f4519439d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 08/04/2025 05:33:28.792 7f4519439d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 08/04/2025 05:33:28.792 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/04/2025 05:33:28.794 7f4519439d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3751-1. 08/04/2025 05:33:28.797 7f45163826d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/04/2025 05:33:28.852 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3751-1.'] 08/04/2025 05:33:28.865 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/04/2025 05:33:28.868 7f45163826d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/04/2025 05:33:28.892 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3751-1.'] 08/04/2025 05:33:29.309 7f4519439d48 [contrail-device-manager] [DEBUG]: Fetching all containers 08/04/2025 05:33:29.420 7f4519439d48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 08/04/2025 05:33:29.612 7f45106d6630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:29.613 7f45106d6630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:29.614 7f45106d6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:29.614 7f45106d6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:29.614 7f45106d6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:33:29.614 7f45106d6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:29.614 7f45106d6630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:29.615 7f45106d6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:29.619 7f4519439d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3751-1..config_queue 08/04/2025 05:33:29.620 7f4519439d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3751-1..tftp_queue 08/04/2025 05:33:29.908 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/04/2025 05:33:29.913 7f45163826d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/04/2025 05:33:30.016 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3751-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3751-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3751-1..tftp_queue'] 08/04/2025 05:33:30.121 7f4519439d48 [contrail-device-manager] [WARNING]: Process 136 prepared to run in Full mode... 08/04/2025 05:33:30.121 7f4519439d48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 08/04/2025 05:33:30.121 7f4519439d48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 08/04/2025 05:33:30.124 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 08/04/2025 05:33:30.124 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 08/04/2025 05:33:30.125 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 08/04/2025 05:33:30.125 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 08/04/2025 05:33:30.126 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 08/04/2025 05:33:30.126 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 08/04/2025 05:33:30.130 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 08/04/2025 05:33:30.131 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 08/04/2025 05:33:30.133 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 08/04/2025 05:33:30.133 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 08/04/2025 05:33:30.133 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 08/04/2025 05:33:30.133 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 08/04/2025 05:33:30.133 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 08/04/2025 05:33:30.133 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 08/04/2025 05:33:30.134 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 08/04/2025 05:33:30.135 7f4519439d48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 08/04/2025 05:33:30.135 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 08/04/2025 05:33:30.135 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 08/04/2025 05:33:30.135 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 08/04/2025 05:33:30.135 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 08/04/2025 05:33:30.135 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 08/04/2025 05:33:30.136 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 08/04/2025 05:33:30.136 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 08/04/2025 05:33:30.136 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 08/04/2025 05:33:30.136 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 08/04/2025 05:33:30.136 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 08/04/2025 05:33:30.137 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 08/04/2025 05:33:30.138 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 08/04/2025 05:33:30.138 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 08/04/2025 05:33:30.138 7f4519439d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 08/04/2025 05:33:30.141 7f4519439d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 08/04/2025 05:33:30.207 7f45106618d0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 08/04/2025 05:33:30.208 7f45106618d0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 08/04/2025 05:33:30.208 7f450b685780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:30.208 7f450b685780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/04/2025 05:33:30.208 7f450b685780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:30.221 7f450b5e2828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:30.221 7f450b5e2828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/04/2025 05:33:30.221 7f450b5e2828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:31.046 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 08/04/2025 05:33:31.049 7f45163826d8 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 08/04/2025 05:33:31.116 7f45163826d8 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-3751-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3751-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-3751-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 08/04/2025 05:33:33.614 7f450b63db70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:33.615 7f450b63db70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:33.616 7f450b63db70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:33.617 7f450b63db70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:33.617 7f450b63db70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:37.616 7f450b515e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:37.618 7f450b515e10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:37.618 7f450b515e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:37.618 7f450b515e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:37.618 7f450b515e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/04/2025 05:33:37.618 7f450b515e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:37.618 7f450b515e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:37.619 7f450b515e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:39.486 7f450b506eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:39.486 7f450b506eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:39.486 7f450b506eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:41.618 7f450b6bde10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:41.620 7f450b6bde10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:41.620 7f450b6bde10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:41.620 7f450b6bde10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:41.620 7f450b6bde10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:33:41.620 7f450b6bde10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:41.620 7f450b6bde10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:41.621 7f450b6bde10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:45.621 7f450b5c7c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:45.623 7f450b5c7c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:45.623 7f450b5c7c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:45.623 7f450b5c7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:45.624 7f450b5c7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:45.625 7f450b5c7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:45.625 7f450b5c7c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:49.624 7f450b6ab240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:49.625 7f450b6ab240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:49.625 7f450b6ab240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:49.625 7f450b6ab240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:49.625 7f450b6ab240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:33:49.625 7f450b6ab240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:49.625 7f450b6ab240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:49.626 7f450b6ab240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:53.626 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:53.627 7f4511687588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:53.628 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:53.629 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:53.629 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:57.629 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:33:57.630 7f4511687588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:33:57.630 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:33:57.630 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:57.630 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:33:57.631 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:01.631 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:01.632 7f4511687588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:01.632 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:01.632 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:01.632 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:01.633 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:05.633 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:05.634 7f4511687588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:05.634 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:05.634 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:05.634 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:05.634 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:05.634 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:05.635 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:05.635 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:05.635 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:05.635 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:05.635 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:05.638 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:05.638 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:09.634 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:09.635 7f4511687588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:09.635 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:09.635 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:09.636 7f4511687588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:13.636 7f45107d4cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:13.637 7f45107d4cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:13.638 7f45107d4cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:13.639 7f45107d4cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:13.639 7f45107d4cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:17.638 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:17.640 7f450b5fc8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:17.640 7f450b5fc8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:17.640 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:17.640 7f450b5fc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:17.640 7f450b5fc8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:17.640 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:17.641 7f450b5fc8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:21.641 7f450b5fceb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:21.642 7f450b5fceb8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:21.642 7f450b5fceb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:21.642 7f450b5fceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:21.642 7f450b5fceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:21.642 7f450b5fceb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:21.642 7f450b5fceb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:21.643 7f450b5fceb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:25.642 7f450b6bcac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:25.644 7f450b6bcac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:25.645 7f450b6bcac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:25.645 7f450b6bcac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:25.645 7f450b6bcac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:25.645 7f450b6bcac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:25.645 7f450b6bcac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:25.645 7f450b6bcac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:29.651 7f450b6bc6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:29.653 7f450b6bc6d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:29.653 7f450b6bc6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:29.653 7f450b6bc6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:29.653 7f450b6bc6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:29.653 7f450b6bc6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:29.653 7f450b6bc6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:29.654 7f450b6bc6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:33.653 7f450b5f6c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:33.655 7f450b5f6c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:33.655 7f450b5f6c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:33.655 7f450b5f6c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:33.656 7f450b5f6c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:37.656 7f4510770198 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:37.657 7f4510770198 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:37.657 7f4510770198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:37.657 7f4510770198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:37.657 7f4510770198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:37.658 7f4510770198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:41.658 7f450b6b3f60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:41.659 7f450b6b3f60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:41.660 7f450b6b3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:41.661 7f450b6b3f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:41.661 7f450b6b3f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:41.661 7f450b6b3f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:45.660 7f450b5fccc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:45.662 7f450b5fccc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:45.663 7f450b5fccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:45.664 7f450b5fccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:45.664 7f450b5fccc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:45.664 7f450b5fccc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:45.664 7f450b5fccc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:45.664 7f450b5fccc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:49.664 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:49.667 7f450b5fc8d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:49.667 7f450b5fc8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:49.667 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:49.667 7f450b5fc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:49.667 7f450b5fc8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:49.667 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:49.668 7f450b5fc8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:53.668 7f450b5fc240 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:53.669 7f450b5fc240 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:53.669 7f450b5fc240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:53.669 7f450b5fc240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:53.669 7f450b5fc240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:34:53.669 7f450b5fc240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:53.669 7f450b5fc240 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:53.670 7f450b5fc240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:54.994 7f4519439d48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 08/04/2025 05:34:54.994 7f4519439d48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 08/04/2025 05:34:54.994 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/04/2025 05:34:54.994 7f450b5e5a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:54.995 7f450b5e5a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:34:54.995 7f450b5e5a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:54.995 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/04/2025 05:34:55.032 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/04/2025 05:34:55.034 7f450b4be978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:55.034 7f450b4be978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 08/04/2025 05:34:55.034 7f450b4be978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:55.041 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/04/2025 05:34:55.042 7f450b4be780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:55.042 7f450b4be780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:34:55.042 7f450b4be780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:55.066 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:34:55.789 7f4519439d48 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 08/04/2025 05:34:55.793 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:34:55.800 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:34:56.046 7f4519439d48 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 08/04/2025 05:34:56.051 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:34:56.051 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/04/2025 05:34:56.132 7f4519439d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/04/2025 05:34:56.148 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:34:56.909 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 08/04/2025 05:34:56.913 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:34:56.919 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:34:57.670 7f450b477780 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:34:57.671 7f450b477780 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:34:57.671 7f450b477780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:34:57.671 7f450b477780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:57.672 7f450b477780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:34:57.672 7f450b477780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:34:57.672 7f450b477780 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:34:57.672 7f450b477780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:34:57.672 7f450b477780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:57.672 7f450b477780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:34:57.673 7f450b477780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:57.673 7f450b477780 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:57.673 7f450b477780 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:34:57.673 7f450b477780 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:34:58.866 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:34:58.871 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:34:58.879 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:34:59.843 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 08/04/2025 05:34:59.850 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:34:59.856 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:35:00.771 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:35:00.775 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:35:00.782 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:35:01.671 7f450b58cf60 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:35:01.673 7f450b58cf60 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:35:01.673 7f450b58cf60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:35:01.673 7f450b58cf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:01.674 7f450b58cf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:35:01.674 7f450b58cf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:01.674 7f450b58cf60 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:35:01.675 7f450b58cf60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:01.920 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 08/04/2025 05:35:01.925 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:35:01.931 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:35:02.829 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:35:02.833 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:35:02.840 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:35:04.001 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 08/04/2025 05:35:04.006 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:35:04.016 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 08/04/2025 05:35:05.673 7f450b5f6d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:35:05.674 7f450b5f6d68 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:35:05.674 7f450b5f6d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:35:05.674 7f450b5f6d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:35:05.675 7f450b5f6d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.730 7f4519439d48 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 08/04/2025 05:35:05.736 7f4519439d48 [contrail-device-manager] [WARNING]: Zookeeper lock released. 08/04/2025 05:35:05.736 7f450b5bbe10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.736 7f450b5bbe10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:35:05.736 7f450b5bbe10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.785 7f4519439d48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 08/04/2025 05:35:05.785 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 08/04/2025 05:35:05.786 7f450b478d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.786 7f450b478d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:35:05.786 7f450b478d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.786 7f450b478d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.787 7f450b478d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:35:05.787 7f450b478d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.792 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/04/2025 05:35:05.793 7f450b478ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.793 7f450b478ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:35:05.793 7f450b478ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.874 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/04/2025 05:35:05.874 7f450b406b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.874 7f450b406b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:35:05.874 7f450b406b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.877 7f4519439d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/04/2025 05:35:05.877 7f450b47c390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.877 7f450b47c390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:35:05.877 7f450b47c390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.879 7f450b47ccc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 08/04/2025 05:35:05.879 7f450b475438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.879 7f450b475438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 08/04/2025 05:35:05.879 7f450b475438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.884 7f450b47ccc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/04/2025 05:35:05.884 7f450b562f60 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:05.884 7f450b562f60 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:35:05.884 7f450b562f60 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:09.675 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:35:09.676 7f450b6bc588 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 08/04/2025 05:35:09.676 7f450b6bc588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:35:09.676 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:09.676 7f450b6bc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:35:09.676 7f450b6bc588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:09.676 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 08/04/2025 05:35:09.677 7f450b6bc588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 08/04/2025 05:35:09.677 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:09.677 7f450b6bc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 08/04/2025 05:35:09.677 7f450b6bc588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:09.677 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:09.677 7f450b6bc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:35:09.678 7f450b6bc588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:13.677 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:35:13.678 7f450b6bc588 [contrail-device-manager] [INFO]: Session Event: TCP Connected 08/04/2025 05:35:13.678 7f450b6bc588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:35:13.678 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 08/04/2025 05:35:13.679 7f450b6bc588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 08/04/2025 05:35:13.679 7f450b407978 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 08/04/2025 05:35:13.680 7f450b407978 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 08/04/2025 05:35:14.698 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-1.'], 'id_perms': {'created': '2025-08-04T05:35:14.684863', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:14.684863', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10158649836059217780, 'uuid_mslong': 4075265211882360639}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '388e401e-57c1-4f3f-8cfa-c633fbe3cf74'}, 'oper': 'CREATE', 'request-id': 'req-e388f659-2c70-4b69-801c-bee917e02990', 'type': 'analytics_node', 'uuid': '388e401e-57c1-4f3f-8cfa-c633fbe3cf74'} 08/04/2025 05:35:15.130 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-3.'], 'id_perms': {'created': '2025-08-04T05:35:15.115771', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:15.115771', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11525123502796778277, 'uuid_mslong': 5653193014536062994}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4e742d18-4624-4812-9ff1-76c358b52b25'}, 'oper': 'CREATE', 'request-id': 'req-1b18bb07-5c82-48b5-8e02-6c59a5fb3672', 'type': 'analytics_node', 'uuid': '4e742d18-4624-4812-9ff1-76c358b52b25'} 08/04/2025 05:35:15.137 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-2.'], 'id_perms': {'created': '2025-08-04T05:35:15.120275', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:15.120275', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11062648987916508847, 'uuid_mslong': 18408083890066572297}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ff76a6c4-8f02-4409-9986-6ca45f991eaf'}, 'oper': 'CREATE', 'request-id': 'req-9c4f0e71-476a-42d6-8adf-c9a6dcca6704', 'type': 'analytics_node', 'uuid': 'ff76a6c4-8f02-4409-9986-6ca45f991eaf'} 08/04/2025 05:35:24.891 7f450b47bcc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connection Closed 08/04/2025 05:35:24.891 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 08/04/2025 05:35:24.891 7f450b47c4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 08/04/2025 05:35:24.891 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:24.891 7f450b47c4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 08/04/2025 05:35:24.891 7f450b47c4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:24.892 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:24.892 7f450b47c4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 1 pid = 136 http_port = 8096 start_time = 1754285589602575 collector_name = collector_ip = 10.0.0.38:8086 collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.254:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 08/04/2025 05:35:24.892 7f450b47c4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 08/04/2025 05:35:28.892 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 08/04/2025 05:35:28.895 7f450b47c4e0 [contrail-device-manager] [INFO]: Session Event: TCP Connected 08/04/2025 05:35:28.895 7f450b47c4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 08/04/2025 05:35:28.895 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3751-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.38:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.38:9041, 10.0.0.50:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.38:2181, 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.254,10.0.0.38,10.0.0.50:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 08/04/2025 05:35:28.896 7f450b47c4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 08/04/2025 05:35:28.896 7f450b477048 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 08/04/2025 05:35:28.897 7f450b477048 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 08/04/2025 05:35:28.899 7f450b4772e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 08/04/2025 05:35:28.900 7f450b4772e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 08/04/2025 05:35:28.900 7f450b39ce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 08/04/2025 05:35:28.900 7f450b39ce10 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 08/04/2025 05:35:28.900 7f450b39ce10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/04/2025 05:35:28.900 7f450b39ce10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 08/04/2025 05:35:28.900 7f450b39ce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 08/04/2025 05:35:28.901 7f450b580cc0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 08/04/2025 05:35:28.901 7f450b580cc0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 08/04/2025 05:35:30.241 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-2.'], 'id_perms': {'created': '2025-08-04T05:35:30.223790', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:30.223790', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12936201493965367751, 'uuid_mslong': 17725928714009067833}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f5ff263f-d868-4139-b386-9eb846aee9c7'}, 'oper': 'CREATE', 'request-id': 'req-a86ee0aa-2117-4b57-9602-6361df408604', 'type': 'analytics_alarm_node', 'uuid': 'f5ff263f-d868-4139-b386-9eb846aee9c7'} 08/04/2025 05:35:30.262 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-3.'], 'id_perms': {'created': '2025-08-04T05:35:30.223991', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:30.223991', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9908749666819188954, 'uuid_mslong': 8394709318441583413}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '747fffa5-e65f-4335-8982-f352bb0728da'}, 'oper': 'CREATE', 'request-id': 'req-cc7b5469-e7c5-404e-9dbe-db675fe2dc01', 'type': 'analytics_alarm_node', 'uuid': '747fffa5-e65f-4335-8982-f352bb0728da'} 08/04/2025 05:35:30.285 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-1.'], 'id_perms': {'created': '2025-08-04T05:35:30.246356', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:30.246356', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11612227181113404213, 'uuid_mslong': 12154142074089717793}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a8ac2fd2-2bbf-4021-a126-eb18d5666b35'}, 'oper': 'CREATE', 'request-id': 'req-855b50de-a781-46ea-b354-21c7307da53d', 'type': 'analytics_alarm_node', 'uuid': 'a8ac2fd2-2bbf-4021-a126-eb18d5666b35'} 08/04/2025 05:35:35.240 7f450b4772e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001296 08/04/2025 05:35:35.241 7f450b4772e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 08/04/2025 05:35:35.241 7f450b44ff60 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 08/04/2025 05:35:35.242 7f450b44ff60 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 08/04/2025 05:35:35.242 7f450b44ff60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/04/2025 05:35:35.242 7f450b44ff60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 08/04/2025 05:35:39.148 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-1.'], 'id_perms': {'created': '2025-08-04T05:35:39.140142', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:39.140142', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11567931752675402648, 'uuid_mslong': 2664857945412749820}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '24fb7a2e-ad34-49fc-a089-8ca3b82de398'}, 'oper': 'CREATE', 'request-id': 'req-ac0308f3-fada-4ef3-9998-334d69dc872b', 'type': 'analytics_snmp_node', 'uuid': '24fb7a2e-ad34-49fc-a089-8ca3b82de398'} 08/04/2025 05:35:39.517 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.38', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-2.'], 'id_perms': {'created': '2025-08-04T05:35:39.509775', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:39.509775', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12229619979561996894, 'uuid_mslong': 18244031817086879069}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fd2fd246-d063-455d-a9b8-56937bd0125e'}, 'oper': 'CREATE', 'request-id': 'req-07e08a72-dfee-4257-955a-d71b2b653801', 'type': 'analytics_snmp_node', 'uuid': 'fd2fd246-d063-455d-a9b8-56937bd0125e'} 08/04/2025 05:35:39.535 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3751-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3751-3.'], 'id_perms': {'created': '2025-08-04T05:35:39.524107', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:35:39.524107', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13682584776323720058, 'uuid_mslong': 7660697228506121627}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6a50438a-8ced-459b-bde2-4e5c51f1637a'}, 'oper': 'CREATE', 'request-id': 'req-4bfdb665-a147-4ca3-8e62-8ca20bbd5db9', 'type': 'analytics_snmp_node', 'uuid': '6a50438a-8ced-459b-bde2-4e5c51f1637a'} 08/04/2025 05:37:18.664 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-f74a93ab-147d-44d9-9cb8-60d53e40a468', 'type': 'global_system_config', 'uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc'} 08/04/2025 05:37:18.725 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-2bed5787-595e-4e82-a014-be14c8a172ff', 'type': 'global_system_config', 'uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc'} 08/04/2025 05:37:18.796 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3751-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-2'], 'id_perms': {'created': '2025-08-04T05:37:18.788557', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:37:18.788557', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13680716201794495804, 'uuid_mslong': 7938535366550045353}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6e2b57db-1017-42a9-bddb-aae6e8e4653c', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-c390adc8-1559-42fc-8917-a3d13696a700', 'type': 'virtual_router', 'uuid': '6e2b57db-1017-42a9-bddb-aae6e8e4653c'} 08/04/2025 05:37:18.862 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-3751-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-1'], 'id_perms': {'created': '2025-08-04T05:37:18.853920', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:37:18.853920', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13268612815396335242, 'uuid_mslong': 5389383866588807543}}, 'parent_type': 'global-system-config', 'parent_uuid': '11a4bc4d-e0a4-4983-a454-4db950ce82dc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4acaf012-8e3f-4177-b823-950e33e4ee8a', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.25'}, 'oper': 'CREATE', 'request-id': 'req-3cf28f01-9afc-4985-9251-016377586cac', 'type': 'virtual_router', 'uuid': '4acaf012-8e3f-4177-b823-950e33e4ee8a'} 08/04/2025 05:37:19.144 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-2', 'vhost0'], 'id_perms': {'created': '2025-08-04T05:37:19.121045', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:37:19.121045', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9516670177644273124, 'uuid_mslong': 16187584975188475978}}, 'parent_type': 'virtual-router', 'parent_uuid': '6e2b57db-1017-42a9-bddb-aae6e8e4653c', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e0a5dac4-f98f-404a-8412-011ae70a5de4', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:e0:a5:da:c4:f9']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'ddf5440b-5c83-4306-a067-ddcc019a42ac'}]}, 'oper': 'CREATE', 'request-id': 'req-18008dca-107b-40fa-b4ab-2758bcc1411b', 'type': 'virtual_machine_interface', 'uuid': 'e0a5dac4-f98f-404a-8412-011ae70a5de4'} 08/04/2025 05:37:19.185 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-18008dca-107b-40fa-b4ab-2758bcc1411b', 'type': 'virtual_machine_interface', 'uuid': 'e0a5dac4-f98f-404a-8412-011ae70a5de4'} 08/04/2025 05:37:19.253 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-1', 'vhost0'], 'id_perms': {'created': '2025-08-04T05:37:19.211840', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-08-04T05:37:19.211840', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10992257975000560388, 'uuid_mslong': 3043818525100426181}}, 'parent_type': 'virtual-router', 'parent_uuid': '4acaf012-8e3f-4177-b823-950e33e4ee8a', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2a3dd0d2-5d5d-43c5-988c-5863875cf304', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:2a:3d:d0:d2:5d']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'ddf5440b-5c83-4306-a067-ddcc019a42ac'}]}, 'oper': 'CREATE', 'request-id': 'req-6789f329-0f27-4053-a522-91e4fd2173e2', 'type': 'virtual_machine_interface', 'uuid': '2a3dd0d2-5d5d-43c5-988c-5863875cf304'} 08/04/2025 05:37:19.292 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-3751-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-6789f329-0f27-4053-a522-91e4fd2173e2', 'type': 'virtual_machine_interface', 'uuid': '2a3dd0d2-5d5d-43c5-988c-5863875cf304'} 08/04/2025 05:37:19.697 7f450b47c390 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-1b0e7d0b-b618-4c98-8899-43e178f35564', 'type': 'access_control_list', 'uuid': '58bb82b0-ed92-4342-bcdb-cf4e38d1b816'} 08/04/2025 05:37:35.098 7f450b3982e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/04/2025 05:38:13.472 7f450b469240 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 08/04/2025 05:39:19.969 7f450b478d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map