02/06/2025 01:50:40.975 7fd9481afe88 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 02/06/2025 01:50:40.975 7fd9481afe88 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 02/06/2025 01:50:40.975 7fd9481afe88 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 02/06/2025 01:50:40.975 7fd93f81db70 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 02/06/2025 01:50:40.975 7fd93f81db70 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 135 start_time = 1738806640976198 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, 10.0.0.38:8086, 10.0.0.41:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 02/06/2025 01:50:40.976 7fd93f81db70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.028 7fd9481afe88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//', 'pyamqp://guest:guest@10.0.0.41:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//'] 02/06/2025 01:50:41.038 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 02/06/2025 01:50:41.038 7fd93f7c9198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.038 7fd93f7c9198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/06/2025 01:50:41.038 7fd93f7c9198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.038 7fd93f7c9198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.039 7fd93f7c9198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/06/2025 01:50:41.039 7fd93f7c9198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.041 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/06/2025 01:50:41.042 7fd93f688b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.042 7fd93f688b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/06/2025 01:50:41.042 7fd93f688b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.115 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 02/06/2025 01:50:41.116 7fd93f7c9438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.116 7fd93f7c9438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/06/2025 01:50:41.116 7fd93f7c9438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.118 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/06/2025 01:50:41.118 7fd93f7c9198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.119 7fd93f7c9198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/06/2025 01:50:41.119 7fd93f7c9198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.182 7fd9481afe88 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 02/06/2025 01:50:41.183 7fd9481afe88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/06/2025 01:50:41.183 7fd9481afe88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/06/2025 01:50:41.183 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/06/2025 01:50:41.183 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 02/06/2025 01:50:41.184 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/06/2025 01:50:41.184 7fd93f7c4e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 02/06/2025 01:50:41.184 7fd93f7c4e10 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 02/06/2025 01:50:41.185 7fd93f5e9438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.185 7fd93f5e9438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.41:9041, 10.0.0.38:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 02/06/2025 01:50:41.185 7fd93f5e9438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.188 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/06/2025 01:50:41.190 7fd94508e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/06/2025 01:50:41.191 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 02/06/2025 01:50:41.191 7fd93f7c4e10 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 02/06/2025 01:50:41.211 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/06/2025 01:50:41.213 7fd93f57ae10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:41.213 7fd93f57ae10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.41:9041, 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.41:2181, 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 02/06/2025 01:50:41.213 7fd93f57ae10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/06/2025 01:50:44.976 7fd93f620eb8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/06/2025 01:50:44.977 7fd93f620eb8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 02/06/2025 01:50:44.977 7fd93f620eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/06/2025 01:50:44.977 7fd93f620eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/06/2025 01:50:44.977 7fd93f620eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.41:5673, 10.0.0.38:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.41:9041, 10.0.0.38:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.41:2181, 10.0.0.38:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 02/06/2025 01:50:44.977 7fd93f620eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/06/2025 01:50:44.977 7fd93f620eb8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 02/06/2025 01:50:44.978 7fd93f620eb8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 02/06/2025 01:50:44.978 7fd93f620eb8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 02/06/2025 01:50:44.978 7fd93f620eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 02/06/2025 01:50:44.978 7fd93f620eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 02/06/2025 01:50:44.978 7fd942523390 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/06/2025 01:50:44.979 7fd942523390 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/06/2025 01:50:44.982 7fd93f81d8d0 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001017 02/06/2025 01:50:44.983 7fd93f81d8d0 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 02/06/2025 01:50:44.983 7fd9403b2780 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 02/06/2025 01:50:44.983 7fd9403b2780 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 02/06/2025 01:50:44.983 7fd9403b2780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 02/06/2025 01:50:44.983 7fd9403b2780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 02/06/2025 01:50:44.984 7fd9403b2780 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 02/06/2025 01:50:44.984 7fd942523390 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/06/2025 01:50:44.986 7fd942523390 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/06/2025 01:50:56.307 7fd9481afe88 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: module 'gevent._gevent_cevent' has no attribute 'Timeout' 02/06/2025 01:51:06.308 7fd9481afe88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/06/2025 01:51:06.309 7fd9481afe88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/06/2025 01:51:06.309 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/06/2025 01:51:06.310 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/06/2025 01:51:06.344 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/06/2025 01:51:06.353 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/06/2025 01:51:06.359 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/06/2025 01:51:06.375 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/06/2025 01:51:06.379 7fd9481afe88 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 02/06/2025 01:51:06.474 7fd9481afe88 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Feb 6 01:51:06 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 02/06/2025 01:51:16.475 7fd9481afe88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/06/2025 01:51:16.476 7fd9481afe88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/06/2025 01:51:16.476 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/06/2025 01:51:16.478 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/06/2025 01:51:16.571 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/06/2025 01:51:16.585 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/06/2025 01:51:16.616 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/06/2025 01:51:16.634 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/06/2025 01:51:16.636 7fd9481afe88 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 02/06/2025 01:51:16.643 7fd9481afe88 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Feb 6 01:51:16 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 02/06/2025 01:51:26.644 7fd9481afe88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/06/2025 01:51:26.644 7fd9481afe88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/06/2025 01:51:26.644 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/06/2025 01:51:26.646 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/06/2025 01:51:26.690 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/06/2025 01:51:26.705 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/06/2025 01:51:26.714 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/06/2025 01:51:26.740 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/06/2025 01:51:26.821 7fd9481afe88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 02/06/2025 01:51:26.821 7fd9481afe88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 02/06/2025 01:51:26.822 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/06/2025 01:51:26.822 7fd9481afe88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2696-1. 02/06/2025 01:51:26.833 7fd94508e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/06/2025 01:51:26.848 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2696-1.'] 02/06/2025 01:51:26.858 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/06/2025 01:51:26.871 7fd94508e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/06/2025 01:51:26.889 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2696-1.'] 02/06/2025 01:51:27.349 7fd9481afe88 [contrail-device-manager] [DEBUG]: Fetching all containers 02/06/2025 01:51:27.863 7fd9481afe88 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 02/06/2025 01:51:28.280 7fd9481afe88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2696-1..config_queue 02/06/2025 01:51:28.280 7fd9481afe88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2696-1..tftp_queue 02/06/2025 01:51:28.781 7fd9481afe88 [contrail-device-manager] [WARNING]: Process 135 prepared to run in Full mode... 02/06/2025 01:51:28.781 7fd9481afe88 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 02/06/2025 01:51:28.781 7fd9481afe88 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 02/06/2025 01:51:28.785 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 02/06/2025 01:51:28.785 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 02/06/2025 01:51:28.786 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 02/06/2025 01:51:28.786 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 02/06/2025 01:51:28.787 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 02/06/2025 01:51:28.787 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 02/06/2025 01:51:28.797 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 02/06/2025 01:51:28.797 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 02/06/2025 01:51:28.797 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 02/06/2025 01:51:28.797 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 02/06/2025 01:51:28.797 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 02/06/2025 01:51:28.798 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 02/06/2025 01:51:28.798 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 02/06/2025 01:51:28.799 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 02/06/2025 01:51:28.799 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 02/06/2025 01:51:28.799 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 02/06/2025 01:51:28.800 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 02/06/2025 01:51:28.800 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 02/06/2025 01:51:28.800 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 02/06/2025 01:51:28.803 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 02/06/2025 01:51:28.803 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 02/06/2025 01:51:28.804 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 02/06/2025 01:51:28.805 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 02/06/2025 01:51:28.807 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 02/06/2025 01:51:28.807 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 02/06/2025 01:51:28.807 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 02/06/2025 01:51:28.807 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 02/06/2025 01:51:28.807 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 02/06/2025 01:51:28.807 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 02/06/2025 01:51:28.808 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 02/06/2025 01:51:28.808 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 02/06/2025 01:51:28.808 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 02/06/2025 01:51:28.812 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 02/06/2025 01:51:28.813 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 02/06/2025 01:51:28.813 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 02/06/2025 01:51:28.814 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 02/06/2025 01:51:28.814 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 02/06/2025 01:51:28.814 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 02/06/2025 01:51:28.814 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 02/06/2025 01:51:28.814 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 02/06/2025 01:51:28.814 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 02/06/2025 01:51:28.816 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 02/06/2025 01:51:28.816 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 02/06/2025 01:51:28.816 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 02/06/2025 01:51:28.816 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 02/06/2025 01:51:28.816 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 02/06/2025 01:51:28.816 7fd9481afe88 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 02/06/2025 01:51:28.822 7fd9481afe88 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 02/06/2025 01:51:29.109 7fd93e6f4e10 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 02/06/2025 01:51:29.113 7fd93e6f4e10 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 02/06/2025 01:51:29.362 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/06/2025 01:51:29.383 7fd94508e780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/06/2025 01:51:29.423 7fd94508e780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2696-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2696-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2696-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 02/06/2025 01:52:16.146 7fd9481afe88 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 6ef9ea51-9d8b-4c2b-b3da-1a5786d708f3 02/06/2025 01:52:16.222 7fd9481afe88 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: 8adcb258-3b97-4848-bb11-0c207680a7a1 02/06/2025 01:52:16.496 7fd9481afe88 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/06/2025 01:52:16.497 7fd9481afe88 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/06/2025 01:52:16.497 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/06/2025 01:52:16.501 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/06/2025 01:52:16.569 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/06/2025 01:52:16.581 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/06/2025 01:52:16.604 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/06/2025 01:52:16.696 7fd9481afe88 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/06/2025 01:52:16.755 7fd9481afe88 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//', 'pyamqp://guest:guest@10.0.0.41:5673//', 'pyamqp://guest:guest@10.0.0.38:5673//'] 02/06/2025 01:52:16.756 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 02/06/2025 01:52:16.757 7fd93de074e0 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/06/2025 01:52:16.757 7fd93de074e0 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/06/2025 01:52:16.762 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/06/2025 01:52:16.792 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 02/06/2025 01:52:16.798 7fd9481afe88 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/06/2025 01:52:16.802 7fd93de04cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 02/06/2025 01:52:16.827 7fd93de04cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED