06/18/2025 18:07:10.355 7f1531213c18 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0]
06/18/2025 18:07:10.355 7f1531213c18 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False]
06/18/2025 18:07:10.355 7f1531213c18 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False]
06/18/2025 18:07:10.358 7f1531213c18 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@localhost:5672/']
06/18/2025 18:07:10.358 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down
06/18/2025 18:07:10.359 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f1522fcfc50>
06/18/2025 18:07:10.359 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
06/18/2025 18:07:10.359 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f1523025c88>
06/18/2025 18:07:10.361 7f1531213c18 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql
06/18/2025 18:07:11.373 7f1531213c18 [contrail-device-manager] [WARNING]: Process 12035 prepared to run in Full mode...
06/18/2025 18:07:11.373 7f1531213c18 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing...
06/18/2025 18:07:11.374 7f1531213c18 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:40409
06/18/2025 18:07:11.426 7f1522fe8e10 [contrail-device-manager] [ERROR]: Cannot write http_port 40409 to /tmp/contrail-device-manager.12027.http_port
06/18/2025 18:07:11.426 7f1522fe8e10 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 40409
06/18/2025 18:07:11.435 7f1531213c18 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json
06/18/2025 18:07:11.435 7f1531213c18 [contrail-device-manager] [ERROR]: Unable to load init data
06/18/2025 18:07:11.435 7f1531213c18 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql
06/18/2025 18:07:11.446 7f1531213c18 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@localhost:5672/']
06/18/2025 18:07:11.446 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down
06/18/2025 18:07:11.447 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f1523025d30>
06/18/2025 18:07:11.447 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
06/18/2025 18:07:11.447 7f1531213c18 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f1522f7f7b8>
06/18/2025 18:07:11.447 7f1522f83cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down
06/18/2025 18:07:11.447 7f1522f83cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f1522f7f7b8>
06/18/2025 18:07:15.331 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(b832b6a6-3e7b-44cf-a207-f292dccaa9ec), feature=l3-gateway
06/18/2025 18:07:15.331 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(b832b6a6-3e7b-44cf-a207-f292dccaa9ec), feature=vn-interconnect
06/18/2025 18:07:15.331 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=b832b6a6-3e7b-44cf-a207-f292dccaa9ec, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:15.377 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(b832b6a6-3e7b-44cf-a207-f292dccaa9ec), feature=l3-gateway
06/18/2025 18:07:15.377 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(b832b6a6-3e7b-44cf-a207-f292dccaa9ec), feature=vn-interconnect
06/18/2025 18:07:15.377 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=b832b6a6-3e7b-44cf-a207-f292dccaa9ec, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:15.694 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:15 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_ee92f88b-1d4a-490e-b471-0b08e7e4049b__', '__contrail_lr_internal_vn_ee92f88b-1d4a-490e-b471-0b08e7e4049b__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_ee92f88b-1d4a-490e-b471-0b08e7e4049b__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_ee92f88b-1d4a-490e-b471-0b08e7e4049b__', '__contrail_lr_internal_vn_ee92f88b-1d4a-490e-b471-0b08e7e4049b__'], 'id_perms': {'created': '2025-06-19T01:07:15.669003', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-06-19T01:07:15.669003', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12769034345554271113, 'uuid_mslong': 7034475067756200152}}, 'parent_type': 'virtual-network', 'parent_uuid': 'c4fc23ad-f902-421f-ae59-5b73e958e75f', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '619f79cd-caf3-40d8-b134-b91583865b89'}, 'oper': 'CREATE', 'request-id': 'req-6d99fbbe-1cc0-49a7-81d8-617ed0c2dc7c', 'type': 'routing_instance', 'uuid': '619f79cd-caf3-40d8-b134-b91583865b89'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:15.772 7f1522f83d68 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'ee92f88b-1d4a-490e-b471-0b08e7e4049b'}. This may indicate that the cluster needs a 'nodetool repair'.
06/18/2025 18:07:15.772 7f1522f83d68 [contrail-device-manager] [WARNING]: logical_router uuid ee92f88b-1d4a-490e-b471-0b08e7e4049b update caused NoIdError
06/18/2025 18:07:17.990 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(5b6f90f4-cef8-4e52-a819-79c36d3b2d46), feature=l3-gateway
06/18/2025 18:07:17.990 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(5b6f90f4-cef8-4e52-a819-79c36d3b2d46), feature=vn-interconnect
06/18/2025 18:07:17.990 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=5b6f90f4-cef8-4e52-a819-79c36d3b2d46, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:18.033 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(5b6f90f4-cef8-4e52-a819-79c36d3b2d46), feature=l3-gateway
06/18/2025 18:07:18.034 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(5b6f90f4-cef8-4e52-a819-79c36d3b2d46), feature=vn-interconnect
06/18/2025 18:07:18.034 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=5b6f90f4-cef8-4e52-a819-79c36d3b2d46, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:18.340 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:18 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_6ce63889-da22-4709-a002-7ba16b92e2a0__', '__contrail_lr_internal_vn_6ce63889-da22-4709-a002-7ba16b92e2a0__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_6ce63889-da22-4709-a002-7ba16b92e2a0__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_6ce63889-da22-4709-a002-7ba16b92e2a0__', '__contrail_lr_internal_vn_6ce63889-da22-4709-a002-7ba16b92e2a0__'], 'id_perms': {'created': '2025-06-19T01:07:18.317046', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-06-19T01:07:18.317046', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13297257919049961691, 'uuid_mslong': 9372631042593278861}}, 'parent_type': 'virtual-network', 'parent_uuid': '7aee4757-7b53-4d2b-9903-dce9a542987f', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '82124646-6092-4b8d-b889-59a01fb910db'}, 'oper': 'CREATE', 'request-id': 'req-9da73bba-cf42-4c9f-830e-0305c26163eb', 'type': 'routing_instance', 'uuid': '82124646-6092-4b8d-b889-59a01fb910db'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:18.418 7f1522f83d68 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'6ce63889-da22-4709-a002-7ba16b92e2a0'}. This may indicate that the cluster needs a 'nodetool repair'.
06/18/2025 18:07:18.418 7f1522f83d68 [contrail-device-manager] [WARNING]: logical_router uuid 6ce63889-da22-4709-a002-7ba16b92e2a0 update caused NoIdError
06/18/2025 18:07:20.591 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(4cceefe6-4fc8-408f-8e7c-e28eebd520d4), feature=vn-interconnect
06/18/2025 18:07:20.591 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(4cceefe6-4fc8-408f-8e7c-e28eebd520d4), feature=l3-gateway
06/18/2025 18:07:20.591 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=4cceefe6-4fc8-408f-8e7c-e28eebd520d4, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:20.635 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(4cceefe6-4fc8-408f-8e7c-e28eebd520d4), feature=vn-interconnect
06/18/2025 18:07:20.635 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(4cceefe6-4fc8-408f-8e7c-e28eebd520d4), feature=l3-gateway
06/18/2025 18:07:20.635 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=4cceefe6-4fc8-408f-8e7c-e28eebd520d4, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:20.938 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:20 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_d27f61e6-c7d2-4491-a68d-3168fba53308__', '__contrail_lr_internal_vn_d27f61e6-c7d2-4491-a68d-3168fba53308__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_d27f61e6-c7d2-4491-a68d-3168fba53308__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_d27f61e6-c7d2-4491-a68d-3168fba53308__', '__contrail_lr_internal_vn_d27f61e6-c7d2-4491-a68d-3168fba53308__'], 'id_perms': {'created': '2025-06-19T01:07:20.915214', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-06-19T01:07:20.915214', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11953936575566440451, 'uuid_mslong': 13631070825087978704}}, 'parent_type': 'virtual-network', 'parent_uuid': '4899a3c7-36f2-4f6c-9d9e-c6fc4ccf28f4', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'bd2b4ab1-e58f-4cd0-a5e4-e9fb162fe803'}, 'oper': 'CREATE', 'request-id': 'req-7d7c23f6-0913-42bd-a21f-1188c19b4c2c', 'type': 'routing_instance', 'uuid': 'bd2b4ab1-e58f-4cd0-a5e4-e9fb162fe803'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:21.014 7f1522f83d68 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'d27f61e6-c7d2-4491-a68d-3168fba53308'}. This may indicate that the cluster needs a 'nodetool repair'.
06/18/2025 18:07:21.015 7f1522f83d68 [contrail-device-manager] [WARNING]: logical_router uuid d27f61e6-c7d2-4491-a68d-3168fba53308 update caused NoIdError
06/18/2025 18:07:23.189 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(bb24072e-fab6-42ea-bef5-105f5877b484), feature=l3-gateway
06/18/2025 18:07:23.189 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(bb24072e-fab6-42ea-bef5-105f5877b484), feature=vn-interconnect
06/18/2025 18:07:23.189 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=bb24072e-fab6-42ea-bef5-105f5877b484, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:23.235 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(bb24072e-fab6-42ea-bef5-105f5877b484), feature=l3-gateway
06/18/2025 18:07:23.235 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(bb24072e-fab6-42ea-bef5-105f5877b484), feature=vn-interconnect
06/18/2025 18:07:23.236 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=bb24072e-fab6-42ea-bef5-105f5877b484, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:23.539 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:23 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_c03f5a64-7aa4-4efe-b751-93b6dc404f7b__', '__contrail_lr_internal_vn_c03f5a64-7aa4-4efe-b751-93b6dc404f7b__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_c03f5a64-7aa4-4efe-b751-93b6dc404f7b__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_c03f5a64-7aa4-4efe-b751-93b6dc404f7b__', '__contrail_lr_internal_vn_c03f5a64-7aa4-4efe-b751-93b6dc404f7b__'], 'id_perms': {'created': '2025-06-19T01:07:23.516826', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-06-19T01:07:23.516826', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12538991744405883883, 'uuid_mslong': 6202385245753525090}}, 'parent_type': 'virtual-network', 'parent_uuid': '98121598-c1d7-4902-8867-9e79e23fdf2e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '56134c84-f543-4362-ae03-728e9ede4beb'}, 'oper': 'CREATE', 'request-id': 'req-50f32683-18da-45ce-aa13-684be830c165', 'type': 'routing_instance', 'uuid': '56134c84-f543-4362-ae03-728e9ede4beb'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:23.832 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:23 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_02d665e4-d4d6-46e3-84b2-699e577f5085__', '__contrail_lr_internal_vn_02d665e4-d4d6-46e3-84b2-699e577f5085__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_02d665e4-d4d6-46e3-84b2-699e577f5085__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_02d665e4-d4d6-46e3-84b2-699e577f5085__', '__contrail_lr_internal_vn_02d665e4-d4d6-46e3-84b2-699e577f5085__'], 'id_perms': {'created': '2025-06-19T01:07:23.810241', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-06-19T01:07:23.810241', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11187297848627299274, 'uuid_mslong': 1833658501295588444}}, 'parent_type': 'virtual-network', 'parent_uuid': 'bd2b6d97-7fe5-4e57-a5a7-e7947d4189ec', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '197276b1-1919-445c-9b41-441ed7ec03ca'}, 'oper': 'CREATE', 'request-id': 'req-292c8fbe-9b8d-4599-bc1a-a8c9c06c011b', 'type': 'routing_instance', 'uuid': '197276b1-1919-445c-9b41-441ed7ec03ca'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:23.909 7f1522f83d68 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'c03f5a64-7aa4-4efe-b751-93b6dc404f7b'}. This may indicate that the cluster needs a 'nodetool repair'.
06/18/2025 18:07:23.909 7f1522f83d68 [contrail-device-manager] [WARNING]: logical_router uuid c03f5a64-7aa4-4efe-b751-93b6dc404f7b update caused NoIdError
06/18/2025 18:07:23.962 7f1522f83d68 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'02d665e4-d4d6-46e3-84b2-699e577f5085'}. This may indicate that the cluster needs a 'nodetool repair'.
06/18/2025 18:07:23.962 7f1522f83d68 [contrail-device-manager] [WARNING]: logical_router uuid 02d665e4-d4d6-46e3-84b2-699e577f5085 update caused NoIdError
06/18/2025 18:07:26.242 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(6382bf20-7416-463d-b3fd-fe878e165211), feature=l3-gateway
06/18/2025 18:07:26.242 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(6382bf20-7416-463d-b3fd-fe878e165211), feature=vn-interconnect
06/18/2025 18:07:26.242 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=6382bf20-7416-463d-b3fd-fe878e165211, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:26.286 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(6382bf20-7416-463d-b3fd-fe878e165211), feature=l3-gateway
06/18/2025 18:07:26.286 7f1522f83d68 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(6382bf20-7416-463d-b3fd-fe878e165211), feature=vn-interconnect
06/18/2025 18:07:26.286 7f1522f83d68 [contrail-device-manager] [INFO]: Found ansible plugin pr=6382bf20-7416-463d-b3fd-fe878e165211, role/vendor/product=spine:juniper:qfx10008
06/18/2025 18:07:26.835 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:26 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_e4487f7f-6547-4a6c-824d-89398d177603__', '__contrail_lr_internal_vn_e4487f7f-6547-4a6c-824d-89398d177603__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_e4487f7f-6547-4a6c-824d-89398d177603__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_e4487f7f-6547-4a6c-824d-89398d177603__', '__contrail_lr_internal_vn_e4487f7f-6547-4a6c-824d-89398d177603__'], 'id_perms': {'created': '2025-06-19T01:07:26.809709', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-06-19T01:07:26.809709', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9227114568090430532, 'uuid_mslong': 14980167185143777537}}, 'parent_type': 'virtual-network', 'parent_uuid': 'c7ece7a1-7cbb-4e9b-ba3f-f03fd176be85', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'cfe43eaf-c275-4901-800d-4bcff4dcac44'}, 'oper': 'CREATE', 'request-id': 'req-d5ea6ff7-7a68-4046-907c-c50005714084', 'type': 'routing_instance', 'uuid': 'cfe43eaf-c275-4901-800d-4bcff4dcac44'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:26.841 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:26 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', 'lr-test.test_dm_ansible_dhcp_relay.TestAnsibleDh...DM.test_dhcp_relay_config_push_two_vns_in_network'], 'oper': 'UPDATE', 'request-id': 'req-d5ea6ff7-7a68-4046-907c-c50005714084', 'type': 'logical_router', 'uuid': 'e4487f7f-6547-4a6c-824d-89398d177603'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:26.853 7f1522f83d68 [contrail-device-manager] [ERROR]: AttributeError
Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python
Wed Jun 18 18:07:26 2025
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', 'lr-test.test_dm_ansible_dhcp_relay.TestAnsibleDh...DM.test_dhcp_relay_config_push_two_vns_in_network'], 'oper': 'UPDATE', 'request-id': 'req-24f43f27-f5b7-4cf3-9c20-d4f6b8c8c38b', 'type': 'logical_router', 'uuid': 'e4487f7f-6547-4a6c-824d-89398d177603'})
82 try:
83 self.oper_info = oper_info
84 self.vnc_subscribe_actions()
85
86 except ConnectionError:
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>)
146 (self. obj_type, obj_id, oper))
147 return
148 self.evaluate_dependency()
149
150 def _get_key_from_oper_info(self):
self = <device_manager.dm_amqp.DMAmqpHandle object>
self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>>
/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>)
49 []):
50 pr = PhysicalRouterDM.get(pr_id)
51 if pr is not None:
52 pr.set_config_state()
53 pr.uve_send()
pr = <device_manager.db.PhysicalRouterDM object>
pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>>
/root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True)
1044 self.nc_q.put_nowait(1)
1045 except queue.Full:
1046 self._logger.warn(f"queue is full")
1047
1048 # end
self = <device_manager.db.PhysicalRouterDM object>
self._logger = <device_manager.logger.DeviceManagerLogger object>
self._logger.warn undefined
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
__cause__ = None
__class__ = <class 'AttributeError'>
__context__ = Full()
__delattr__ = <method-wrapper '__delattr__' of AttributeError object>
__dict__ = {}
__dir__ = <built-in method __dir__ of AttributeError object>
__doc__ = 'Attribute not found.'
__eq__ = <method-wrapper '__eq__' of AttributeError object>
__format__ = <built-in method __format__ of AttributeError object>
__ge__ = <method-wrapper '__ge__' of AttributeError object>
__getattribute__ = <method-wrapper '__getattribute__' of AttributeError object>
__gt__ = <method-wrapper '__gt__' of AttributeError object>
__hash__ = <method-wrapper '__hash__' of AttributeError object>
__init__ = <method-wrapper '__init__' of AttributeError object>
__init_subclass__ = <built-in method __init_subclass__ of type object>
__le__ = <method-wrapper '__le__' of AttributeError object>
__lt__ = <method-wrapper '__lt__' of AttributeError object>
__ne__ = <method-wrapper '__ne__' of AttributeError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of AttributeError object>
__reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object>
__repr__ = <method-wrapper '__repr__' of AttributeError object>
__setattr__ = <method-wrapper '__setattr__' of AttributeError object>
__setstate__ = <built-in method __setstate__ of AttributeError object>
__sizeof__ = <built-in method __sizeof__ of AttributeError object>
__str__ = <method-wrapper '__str__' of AttributeError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__suppress_context__ = False
__traceback__ = <traceback object>
args = ("'DeviceManagerLogger' object has no attribute 'warn'",)
with_traceback = <built-in method with_traceback of AttributeError object>
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state
self.nc_q.put_nowait(1)
File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait
File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put
queue.Full
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback
self.vnc_subscribe_actions()
File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions
self.evaluate_dependency()
File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency
pr.set_config_state()
File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state
self._logger.warn(f"queue is full")
AttributeError: 'DeviceManagerLogger' object has no attribute 'warn'
06/18/2025 18:07:27.871 7f1522f83d68 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'e4487f7f-6547-4a6c-824d-89398d177603'}. This may indicate that the cluster needs a 'nodetool repair'.
06/18/2025 18:07:27.871 7f1522f83d68 [contrail-device-manager] [WARNING]: logical_router uuid e4487f7f-6547-4a6c-824d-89398d177603 update caused NoIdError
06/18/2025 18:07:32.544 7f1522832828 [contrail-device-manager] [INFO]: SANDESH: CONNECT TO COLLECTOR: False
06/18/2025 18:07:32.554 7f1522832828 [contrail-device-manager] [INFO]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_INFO]
06/18/2025 18:07:32.555 7f1522832828 [contrail-device-manager] [INFO]: SANDESH: Logging: FILE: [None] -> [device_manager_TestAnsibleCollapsedSpineDM.log]