03/20/2025 01:52:38.162 7f501c3cb888 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/20/2025 01:52:38.162 7f501c3cb888 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/20/2025 01:52:38.162 7f501c3cb888 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/20/2025 01:52:38.162 7f5013a6a390 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.163 7f5013a6a390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.164 7f5013a6a390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1742435558163474 collector_name = collector_ip = collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/20/2025 01:52:38.164 7f5013a6a390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.217 7f501c3cb888 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.42:5673//', 'pyamqp://guest:guest@10.0.0.12:5673//', 'pyamqp://guest:guest@10.0.0.47:5673//'] 03/20/2025 01:52:38.232 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 03/20/2025 01:52:38.232 7f50139bd630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.233 7f50139bd630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/20/2025 01:52:38.233 7f50139bd630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.233 7f50139bd630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.233 7f50139bd630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/20/2025 01:52:38.233 7f50139bd630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.236 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/20/2025 01:52:38.237 7f50139bd2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.237 7f50139bd2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:38.237 7f50139bd2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.398 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/20/2025 01:52:38.398 7f50139bd8d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.398 7f50139bd8d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/20/2025 01:52:38.398 7f50139bd8d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.405 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/20/2025 01:52:38.405 7f50139bd198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.405 7f50139bd198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/20/2025 01:52:38.406 7f50139bd198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.489 7f501c3cb888 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 03/20/2025 01:52:38.490 7f501c3cb888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/20/2025 01:52:38.490 7f501c3cb888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/20/2025 01:52:38.490 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/20/2025 01:52:38.491 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 03/20/2025 01:52:38.491 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/20/2025 01:52:38.491 7f5016205a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 03/20/2025 01:52:38.491 7f5016205a20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 03/20/2025 01:52:38.491 7f5013859240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.491 7f5013859240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/20/2025 01:52:38.492 7f5013859240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.493 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/20/2025 01:52:38.496 7f50192f5780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/20/2025 01:52:38.497 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 03/20/2025 01:52:38.499 7f5016205a20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 03/20/2025 01:52:38.529 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/20/2025 01:52:38.531 7f50137b5630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:38.531 7f50137b5630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/20/2025 01:52:38.531 7f50137b5630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:42.164 7f50221c9630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:52:42.165 7f50221c9630 [contrail-device-manager] [INFO]: Session Event: TCP Connected 03/20/2025 01:52:42.166 7f50221c9630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:52:42.166 7f50221c9630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:52:42.166 7f50221c9630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/20/2025 01:52:42.166 7f50221c9630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:52:42.166 7f50221c9630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:52:42.166 7f50221c9630 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 03/20/2025 01:52:42.167 7f50221c9630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:52:42.167 7f50221c9630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:52:42.167 7f50221c9630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:52:42.168 7f5012ef80f0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:52:42.169 7f5012ef80f0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:52:46.880 7f5013a6a0f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:52:46.880 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:52:46.880 7f5014608588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:52:46.881 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:46.881 7f5014608588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Initializing description = >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/20/2025 01:52:46.881 7f5014608588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:46.881 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:46.881 7f5014608588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 1 pid = 136 start_time = 1742435558163474 collector_name = collector_ip = 10.0.0.12:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 01:52:46.881 7f5014608588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:52:50.881 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [INFO]: Session Event: TCP Connected 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:52:50.883 7f5014608588 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 03/20/2025 01:52:50.884 7f5014608588 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:52:50.884 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:52:50.884 7f5014608588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:52:50.884 7f5012ef8198 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:52:50.884 7f5012ef8198 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:52:53.604 7f501c3cb888 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: module 'gevent._gevent_cevent' has no attribute 'Timeout' 03/20/2025 01:53:03.136 7f5012ef8390 [contrail-device-manager] [ERROR]: Session Event: TCP Connection Closed 03/20/2025 01:53:03.136 7f50166ecac8 [contrail-device-manager] [INFO]: Processing event[EvTcpClose] in state[ClientInit] 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpClose] => State[ClientInit] -> State[Idle] 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Initializing description = >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 2 pid = 136 start_time = 1742435558163474 collector_name = collector_ip = 10.0.0.47:8086 collector_list = [ 10.0.0.12:8086, 10.0.0.47:8086, 10.0.0.42:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/20/2025 01:53:03.137 7f50166ecac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.605 7f501c3cb888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/20/2025 01:53:03.606 7f501c3cb888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/20/2025 01:53:03.606 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/20/2025 01:53:03.607 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/20/2025 01:53:03.637 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/20/2025 01:53:03.676 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/20/2025 01:53:03.676 7f50190460f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.676 7f50190460f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Up description = >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose], Database:Cassandra[] connection down >>, ] >> 03/20/2025 01:53:03.676 7f50190460f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.683 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/20/2025 01:53:03.712 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/20/2025 01:53:03.712 7f5012d8e4e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.712 7f5012d8e4e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Up description = >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose] connection down >>, ] >> 03/20/2025 01:53:03.712 7f5012d8e4e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.714 7f501c3cb888 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 03/20/2025 01:53:03.848 7f501c3cb888 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Mar 20 01:53:03 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 03/20/2025 01:53:03.849 7f5012d94ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:03.849 7f5012d94ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.47:8086, ] status = Down description = ClientInit to Idle on EvTcpClose >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Up description = >>, ] description = Collector:Collector[ClientInit to Idle on EvTcpClose], Database:Cassandra[] connection down >>, ] >> 03/20/2025 01:53:03.849 7f5012d94ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/20/2025 01:53:07.139 7f5012d94d68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/20/2025 01:53:07.140 7f5012d94d68 [contrail-device-manager] [INFO]: Session Event: TCP Connected 03/20/2025 01:53:07.140 7f5012d94d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/20/2025 01:53:07.140 7f5012d94d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:07.140 7f5012d94d68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.42:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.42:5673, 10.0.0.12:5673, 10.0.0.47:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.42:9041, 10.0.0.12:9041, 10.0.0.47:9041, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.42:2181, 10.0.0.12:2181, 10.0.0.47:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/20/2025 01:53:07.140 7f5012d94d68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/20/2025 01:53:07.140 7f5012d94d68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/20/2025 01:53:07.141 7f5012d94d68 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 03/20/2025 01:53:07.141 7f5012d94d68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/20/2025 01:53:07.141 7f5012d94d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:07.141 7f5012d94d68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/20/2025 01:53:07.141 7f5012d8e828 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:53:07.142 7f5012d8e828 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:53:07.150 7f5012d8e4e0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 03/20/2025 01:53:07.150 7f5012d8e4e0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/20/2025 01:53:07.151 7f5012ef80f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/20/2025 01:53:07.151 7f5012ef80f0 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 03/20/2025 01:53:07.151 7f5012ef80f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/20/2025 01:53:07.151 7f5012ef80f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/20/2025 01:53:07.152 7f5012ef80f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/20/2025 01:53:07.152 7f5012d8e240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:53:07.154 7f5012d8e240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:53:13.850 7f501c3cb888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/20/2025 01:53:13.850 7f501c3cb888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/20/2025 01:53:13.850 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/20/2025 01:53:13.851 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/20/2025 01:53:13.906 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/20/2025 01:53:13.916 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/20/2025 01:53:13.923 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/20/2025 01:53:13.947 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/20/2025 01:53:13.950 7f501c3cb888 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 03/20/2025 01:53:13.961 7f501c3cb888 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Mar 20 01:53:13 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 03/20/2025 01:53:23.962 7f501c3cb888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/20/2025 01:53:23.962 7f501c3cb888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/20/2025 01:53:23.962 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/20/2025 01:53:23.964 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/20/2025 01:53:24.010 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/20/2025 01:53:24.035 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/20/2025 01:53:24.054 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/20/2025 01:53:24.090 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/20/2025 01:53:24.097 7f501c3cb888 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 03/20/2025 01:53:24.120 7f501c3cb888 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Mar 20 01:53:24 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 03/20/2025 01:53:34.122 7f501c3cb888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/20/2025 01:53:34.123 7f501c3cb888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/20/2025 01:53:34.123 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/20/2025 01:53:34.124 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/20/2025 01:53:34.183 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/20/2025 01:53:34.236 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/20/2025 01:53:34.244 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/20/2025 01:53:34.260 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/20/2025 01:53:34.424 7f501c3cb888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 03/20/2025 01:53:34.424 7f501c3cb888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 03/20/2025 01:53:34.424 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/20/2025 01:53:34.434 7f501c3cb888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2923-1. 03/20/2025 01:53:34.445 7f50192f5780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/20/2025 01:53:34.593 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2923-1.'] 03/20/2025 01:53:34.643 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/20/2025 01:53:34.649 7f50192f5780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/20/2025 01:53:34.712 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2923-1.'] 03/20/2025 01:53:34.953 7f501c3cb888 [contrail-device-manager] [DEBUG]: Fetching all containers 03/20/2025 01:53:35.374 7f501c3cb888 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 03/20/2025 01:53:35.830 7f501c3cb888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2923-1..config_queue 03/20/2025 01:53:35.830 7f501c3cb888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2923-1..tftp_queue 03/20/2025 01:53:36.331 7f501c3cb888 [contrail-device-manager] [WARNING]: Process 136 prepared to run in Full mode... 03/20/2025 01:53:36.331 7f501c3cb888 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 03/20/2025 01:53:36.331 7f501c3cb888 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 03/20/2025 01:53:36.334 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 03/20/2025 01:53:36.334 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 03/20/2025 01:53:36.335 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 03/20/2025 01:53:36.335 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 03/20/2025 01:53:36.335 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 03/20/2025 01:53:36.335 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 03/20/2025 01:53:36.341 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 03/20/2025 01:53:36.341 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 03/20/2025 01:53:36.341 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 03/20/2025 01:53:36.341 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 03/20/2025 01:53:36.342 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 03/20/2025 01:53:36.342 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 03/20/2025 01:53:36.343 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 03/20/2025 01:53:36.343 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 03/20/2025 01:53:36.343 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 03/20/2025 01:53:36.343 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 03/20/2025 01:53:36.343 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 03/20/2025 01:53:36.345 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 03/20/2025 01:53:36.345 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 03/20/2025 01:53:36.353 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 03/20/2025 01:53:36.355 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 03/20/2025 01:53:36.355 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 03/20/2025 01:53:36.355 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 03/20/2025 01:53:36.358 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 03/20/2025 01:53:36.358 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 03/20/2025 01:53:36.358 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 03/20/2025 01:53:36.361 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 03/20/2025 01:53:36.361 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 03/20/2025 01:53:36.361 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 03/20/2025 01:53:36.361 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 03/20/2025 01:53:36.361 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 03/20/2025 01:53:36.361 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 03/20/2025 01:53:36.365 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 03/20/2025 01:53:36.367 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 03/20/2025 01:53:36.367 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 03/20/2025 01:53:36.367 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 03/20/2025 01:53:36.367 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 03/20/2025 01:53:36.369 7f501c3cb888 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 03/20/2025 01:53:36.378 7f501c3cb888 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 03/20/2025 01:53:36.566 7f50129b3d68 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 03/20/2025 01:53:36.566 7f50129b3d68 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 03/20/2025 01:53:36.783 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/20/2025 01:53:36.792 7f50192f5780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/20/2025 01:53:37.150 7f50192f5780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2923-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2923-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2923-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 03/20/2025 01:54:23.706 7f501c3cb888 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 98f5b8a6-3f1b-4967-8a1a-82c848d3cd73 03/20/2025 01:54:23.824 7f501c3cb888 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 826220c5-748a-4986-b4b8-2a6455c02f53 03/20/2025 01:54:24.287 7f501c3cb888 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/20/2025 01:54:24.288 7f501c3cb888 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/20/2025 01:54:24.288 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/20/2025 01:54:24.290 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/20/2025 01:54:24.399 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/20/2025 01:54:24.413 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/20/2025 01:54:24.448 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/20/2025 01:54:24.562 7f501c3cb888 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/20/2025 01:54:24.654 7f501c3cb888 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.42:5673//', 'pyamqp://guest:guest@10.0.0.12:5673//', 'pyamqp://guest:guest@10.0.0.47:5673//'] 03/20/2025 01:54:24.655 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 03/20/2025 01:54:24.655 7f5011fd78d0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/20/2025 01:54:24.656 7f5011fd78d0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/20/2025 01:54:24.661 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/20/2025 01:54:24.700 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/20/2025 01:54:24.704 7f501c3cb888 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/20/2025 01:54:24.708 7f5011fd76d8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/20/2025 01:54:24.714 7f5011fd76d8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED