02/13/2025 02:08:09.340 7f4a2d998d48 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 02/13/2025 02:08:09.340 7f4a2d998d48 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 02/13/2025 02:08:09.340 7f4a2d998d48 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 02/13/2025 02:08:09.341 7f4a250926d8 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 02/13/2025 02:08:09.341 7f4a250926d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 02/13/2025 02:08:09.342 7f4a250926d8 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 02/13/2025 02:08:09.342 7f4a250926d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 02/13/2025 02:08:09.342 7f4a250926d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.342 7f4a250926d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:08:09.342 7f4a250926d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.342 7f4a250926d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.343 7f4a250926d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 138 start_time = 1739412489341913 collector_name = collector_ip = collector_list = [ 10.0.0.12:8086, 10.0.0.46:8086, 10.0.0.35:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 02/13/2025 02:08:09.343 7f4a250926d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.439 7f4a2d998d48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.46:5673//', 'pyamqp://guest:guest@10.0.0.35:5673//', 'pyamqp://guest:guest@10.0.0.12:5673//'] 02/13/2025 02:08:09.467 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 02/13/2025 02:08:09.467 7f4a24fe2e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.468 7f4a24fe2e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/13/2025 02:08:09.468 7f4a24fe2e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.468 7f4a24fe2e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.468 7f4a24fe2e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/13/2025 02:08:09.468 7f4a24fe2e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.473 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/13/2025 02:08:09.474 7f4a24fea6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.474 7f4a24fea6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:08:09.474 7f4a24fea6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.492 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 02/13/2025 02:08:09.493 7f4a24fa5588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.493 7f4a24fa5588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 02/13/2025 02:08:09.493 7f4a24fa5588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.501 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/13/2025 02:08:09.501 7f4a24fa56d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.501 7f4a24fa56d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:08:09.501 7f4a24fa56d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.515 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 02/13/2025 02:08:09.516 7f4a2d998d48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/13/2025 02:08:09.516 7f4a2d998d48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/13/2025 02:08:09.516 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/13/2025 02:08:09.517 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 02/13/2025 02:08:09.517 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/13/2025 02:08:09.517 7f4a24fea8d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 02/13/2025 02:08:09.517 7f4a24fea8d0 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 02/13/2025 02:08:09.517 7f4a24fe26d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.518 7f4a24fe26d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.46:9041, 10.0.0.35:9041, 10.0.0.12:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 02/13/2025 02:08:09.518 7f4a24fe26d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.520 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/13/2025 02:08:09.523 7f4a2a91f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/13/2025 02:08:09.523 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 02/13/2025 02:08:09.523 7f4a24fea8d0 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 02/13/2025 02:08:09.572 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/13/2025 02:08:09.573 7f4a24e382e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.574 7f4a24e382e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.46:9041, 10.0.0.35:9041, 10.0.0.12:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.46:2181, 10.0.0.35:2181, 10.0.0.12:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 02/13/2025 02:08:09.574 7f4a24e382e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.586 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/13/2025 02:08:09.587 7f4a24dfa6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.587 7f4a24dfa6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.46:9041, 10.0.0.35:9041, 10.0.0.12:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.46:2181, 10.0.0.35:2181, 10.0.0.12:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 02/13/2025 02:08:09.587 7f4a24dfa6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.592 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/13/2025 02:08:09.608 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/13/2025 02:08:09.608 7f4a24dc8048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.609 7f4a24dc8048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.46:9041, 10.0.0.35:9041, 10.0.0.12:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.46:2181, 10.0.0.35:2181, 10.0.0.12:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 02/13/2025 02:08:09.609 7f4a24dc8048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.610 7f4a2d998d48 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 02/13/2025 02:08:09.713 7f4a2d998d48 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Feb 13 02:08:09 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 02/13/2025 02:08:09.713 7f4a24dcc198 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:09.714 7f4a24dcc198 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.46:9041, 10.0.0.35:9041, 10.0.0.12:9041, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.46:2181, 10.0.0.35:2181, 10.0.0.12:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 02/13/2025 02:08:09.714 7f4a24dcc198 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 02/13/2025 02:08:13.341 7f4a24dcce10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 02/13/2025 02:08:13.343 7f4a24dcce10 [contrail-device-manager] [INFO]: Session Event: TCP Connected 02/13/2025 02:08:13.343 7f4a24dcce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 02/13/2025 02:08:13.343 7f4a24dcce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:08:13.343 7f4a24dcce10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.12:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.46:5673, 10.0.0.35:5673, 10.0.0.12:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.46:9041, 10.0.0.35:9041, 10.0.0.12:9041, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.46:2181, 10.0.0.35:2181, 10.0.0.12:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 02/13/2025 02:08:13.343 7f4a24dcce10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 02/13/2025 02:08:13.343 7f4a24dcce10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 02/13/2025 02:08:13.344 7f4a24dcce10 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 02/13/2025 02:08:13.344 7f4a24dcce10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 02/13/2025 02:08:13.344 7f4a24dcce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 02/13/2025 02:08:13.344 7f4a24dcce10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 02/13/2025 02:08:13.344 7f4a24dc8240 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/13/2025 02:08:13.345 7f4a24dc8240 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/13/2025 02:08:13.350 7f4a24dccb70 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001016 02/13/2025 02:08:13.350 7f4a24dccb70 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 02/13/2025 02:08:13.351 7f4a25c3aa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 02/13/2025 02:08:13.351 7f4a25c3aa20 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 02/13/2025 02:08:13.351 7f4a25c3aa20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 02/13/2025 02:08:13.351 7f4a25c3aa20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 02/13/2025 02:08:13.352 7f4a25c3aa20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 02/13/2025 02:08:13.352 7f4a24e64c18 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/13/2025 02:08:13.353 7f4a24e64c18 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/13/2025 02:08:19.714 7f4a2d998d48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/13/2025 02:08:19.715 7f4a2d998d48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/13/2025 02:08:19.716 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/13/2025 02:08:19.719 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/13/2025 02:08:19.767 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/13/2025 02:08:19.794 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/13/2025 02:08:19.814 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/13/2025 02:08:19.838 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/13/2025 02:08:19.856 7f4a2d998d48 [contrail-device-manager] [ERROR]: Cassandra connection down. Exception in > 02/13/2025 02:08:19.874 7f4a2d998d48 [contrail-device-manager] [ERROR]: Error while initializing db connection, retrying: Error accessing error, ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}): NoHostAvailable Python 3.6.8: /usr/bin/python3 Thu Feb 13 02:08:19 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in wrapper(*args=(), **kwargs={}) 1029 self._Init_Cluster() 1030 self.start_time = datetime.datetime.now() 1031 return func(*args, **kwargs) 1032 except (connector.InvalidRequest, 1033 connector.cluster.NoHostAvailable, func = > args = () kwargs = {} /usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py in walk(self=, fn=None) 1472 for obj_uuid, obj_col in self._cassandra_driver.get_range( 1473 datastore_api.OBJ_UUID_CF_NAME, 1474 columns=['type', 'fq_name']): 1475 try: 1476 obj_type = json.loads(obj_col['type']) columns undefined /usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py in _Get_Range(self=, cf_name='obj_uuid_table', columns=['type', 'fq_name'], column_count=100000, include_timestamp=False) 884 885 current_key, aggregator = None, [] 886 for row in ses.execute(cql, arg): 887 key, row = row[0], row[1:] 888 row undefined ses = ses.execute = > cql = '\n SELECT blobAsText(key), blobAsText(colu... (textAsBlob(%s), textAsBlob(%s)) ALLOW FILTERING' arg = ['type', 'fq_name'] /cassandra/cluster.py in cassandra.cluster.Session.execute() /cassandra/cluster.py in cassandra.cluster.ResponseFuture.result() NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)}) errors = {: Unavailable('Error from server: code=1000 [Unava...required_replicas\': 2, \'alive_replicas\': 1}',)} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 1031, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/cfgm_common/vnc_cassandra.py", line 1474, in walk columns=['type', 'fq_name']): File "/usr/lib/python3.6/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 886, in _Get_Range for row in ses.execute(cql, arg): File "cassandra/cluster.py", line 2618, in cassandra.cluster.Session.execute File "cassandra/cluster.py", line 4901, in cassandra.cluster.ResponseFuture.result cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: Unavailable('Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={\'consistency\': \'QUORUM\', \'required_replicas\': 2, \'alive_replicas\': 1}',)}) database due to: None 02/13/2025 02:08:29.875 7f4a2d998d48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/13/2025 02:08:29.875 7f4a2d998d48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/13/2025 02:08:29.875 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/13/2025 02:08:29.877 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/13/2025 02:08:30.029 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/13/2025 02:08:30.053 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/13/2025 02:08:30.093 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/13/2025 02:08:30.151 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/13/2025 02:08:30.295 7f4a2d998d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 02/13/2025 02:08:30.295 7f4a2d998d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 02/13/2025 02:08:30.295 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/13/2025 02:08:30.298 7f4a2d998d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2728-1. 02/13/2025 02:08:30.306 7f4a2a91f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/13/2025 02:08:30.411 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2728-1.'] 02/13/2025 02:08:30.453 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/13/2025 02:08:30.458 7f4a2a91f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/13/2025 02:08:30.648 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2728-1.'] 02/13/2025 02:08:30.834 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Fetching all containers 02/13/2025 02:08:31.792 7f4a2d998d48 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 02/13/2025 02:08:32.556 7f4a2d998d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2728-1..config_queue 02/13/2025 02:08:32.557 7f4a2d998d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2728-1..tftp_queue 02/13/2025 02:08:32.688 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/13/2025 02:08:32.705 7f4a2a91f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/13/2025 02:08:33.058 7f4a2d998d48 [contrail-device-manager] [WARNING]: Process 138 prepared to run in Full mode... 02/13/2025 02:08:33.058 7f4a2d998d48 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 02/13/2025 02:08:33.058 7f4a2d998d48 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 02/13/2025 02:08:33.065 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 02/13/2025 02:08:33.065 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 02/13/2025 02:08:33.066 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 02/13/2025 02:08:33.067 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 02/13/2025 02:08:33.068 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 02/13/2025 02:08:33.068 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 02/13/2025 02:08:33.077 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 02/13/2025 02:08:33.077 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 02/13/2025 02:08:33.077 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 02/13/2025 02:08:33.077 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 02/13/2025 02:08:33.078 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 02/13/2025 02:08:33.080 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 02/13/2025 02:08:33.080 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 02/13/2025 02:08:33.080 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 02/13/2025 02:08:33.080 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 02/13/2025 02:08:33.081 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 02/13/2025 02:08:33.081 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 02/13/2025 02:08:33.081 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 02/13/2025 02:08:33.081 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 02/13/2025 02:08:33.082 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 02/13/2025 02:08:33.082 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 02/13/2025 02:08:33.082 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 02/13/2025 02:08:33.082 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 02/13/2025 02:08:33.082 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 02/13/2025 02:08:33.083 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 02/13/2025 02:08:33.084 7f4a2d998d48 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 02/13/2025 02:08:33.087 7f4a2d998d48 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 02/13/2025 02:08:33.639 7f4a2bdbf0f0 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 02/13/2025 02:08:33.644 7f4a2bdbf0f0 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 02/13/2025 02:09:02.795 7f4a2a91f780 [contrail-device-manager] [WARNING]: KombuAmqpClient: Connection error in Kombu amqp consumer greenlet: (0, 0): (541) INTERNAL_ERROR Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cfgm_common/kombu_amqp.py", line 177, in _start_consuming consumers = self._create_consumer_list() File "/usr/lib/python3.6/site-packages/cfgm_common/kombu_amqp.py", line 141, in _create_consumer_list for c in list(self._consumers.values())] File "/usr/lib/python3.6/site-packages/cfgm_common/kombu_amqp.py", line 141, in for c in list(self._consumers.values())] File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 387, in __init__ self.revive(self.channel) File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 409, in revive self.declare() File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 422, in declare queue.declare() File "/usr/local/lib/python3.6/site-packages/kombu/entity.py", line 606, in declare self._create_queue(nowait=nowait, channel=channel) File "/usr/local/lib/python3.6/site-packages/kombu/entity.py", line 615, in _create_queue self.queue_declare(nowait=nowait, passive=False, channel=channel) File "/usr/local/lib/python3.6/site-packages/kombu/entity.py", line 650, in queue_declare nowait=nowait, File "/usr/local/lib/python3.6/site-packages/amqp/channel.py", line 1163, in queue_declare spec.Queue.DeclareOk, returns_tuple=True, File "/usr/local/lib/python3.6/site-packages/amqp/abstract_channel.py", line 99, in wait self.connection.drain_events(timeout=timeout) File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 526, in drain_events while not self.blocking_read(timeout): File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 532, in blocking_read return self.on_inbound_frame(frame) File "/usr/local/lib/python3.6/site-packages/amqp/method_framing.py", line 53, in on_frame callback(channel, method_sig, buf, None) File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 539, in on_inbound_method method_sig, payload, content, File "/usr/local/lib/python3.6/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method listener(*args) File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 669, in _on_close (class_id, method_id), ConnectionError) amqp.exceptions.InternalError: (0, 0): (541) INTERNAL_ERROR 02/13/2025 02:09:02.896 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/13/2025 02:09:02.945 7f4a2a91f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/13/2025 02:09:03.270 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2728-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2728-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2728-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 02/13/2025 02:09:03.329 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 02/13/2025 02:09:03.337 7f4a2a91f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 02/13/2025 02:09:03.426 7f4a2a91f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2728-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2728-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2728-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 02/13/2025 02:09:29.797 7f4a2d998d48 [contrail-device-manager] [ERROR]: network IPAM 'ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_ipam_ipv6_link_local'] already exists with uuid: 625d7040-bf89-459e-bf43-15fcd4be6ab5 02/13/2025 02:09:29.885 7f4a2d998d48 [contrail-device-manager] [ERROR]: virtual network '_internal_vn_ipv6_link_local' already exists or other conflict: ['default-domain', 'default-project', '_internal_vn_ipv6_link_local'] already exists with uuid: ea186bbe-cd34-4af5-a3f7-41d5f6386e6c 02/13/2025 02:09:30.253 7f4a2d998d48 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 02/13/2025 02:09:30.253 7f4a2d998d48 [contrail-device-manager] [WARNING]: initializing cassandra cluster 02/13/2025 02:09:30.253 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 02/13/2025 02:09:30.256 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 02/13/2025 02:09:30.352 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 02/13/2025 02:09:30.367 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 02/13/2025 02:09:30.395 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 02/13/2025 02:09:30.511 7f4a2d998d48 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 02/13/2025 02:09:30.634 7f4a2d998d48 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.46:5673//', 'pyamqp://guest:guest@10.0.0.35:5673//', 'pyamqp://guest:guest@10.0.0.12:5673//'] 02/13/2025 02:09:30.635 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 02/13/2025 02:09:30.636 7f4a1f7526d8 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 02/13/2025 02:09:30.637 7f4a1f7526d8 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 02/13/2025 02:09:30.642 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 02/13/2025 02:09:30.681 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 02/13/2025 02:09:30.686 7f4a2d998d48 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 02/13/2025 02:09:30.688 7f4a1f7c2eb8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 02/13/2025 02:09:30.697 7f4a1f7c2eb8 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED