05/04/2025 18:06:17.396 7f0be7792198 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 05/04/2025 18:06:17.396 7f0be7792198 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 05/04/2025 18:06:17.396 7f0be7792198 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 05/04/2025 18:06:17.399 7f0be7792198 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@localhost:5672/'] 05/04/2025 18:06:17.400 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/04/2025 18:06:17.400 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f0be7683780> 05/04/2025 18:06:17.400 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/04/2025 18:06:17.400 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f0be76f8c50> 05/04/2025 18:06:17.401 7f0be7792198 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/04/2025 18:06:18.411 7f0be7792198 [contrail-device-manager] [WARNING]: Process 12073 prepared to run in Full mode... 05/04/2025 18:06:18.411 7f0be7792198 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 05/04/2025 18:06:18.412 7f0be7792198 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:39033 05/04/2025 18:06:18.429 7f0be77d1f60 [contrail-device-manager] [ERROR]: Cannot write http_port 39033 to /tmp/contrail-device-manager.12063.http_port 05/04/2025 18:06:18.429 7f0be77d1f60 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 39033 05/04/2025 18:06:18.439 7f0be7792198 [contrail-device-manager] [ERROR]: predef payloads file does not exist: /opt/contrail/fabric_ansible_playbooks/conf/predef_payloads.json 05/04/2025 18:06:18.439 7f0be7792198 [contrail-device-manager] [ERROR]: Unable to load init data 05/04/2025 18:06:18.439 7f0be7792198 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 05/04/2025 18:06:18.445 7f0be7792198 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@localhost:5672/'] 05/04/2025 18:06:18.445 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 05/04/2025 18:06:18.445 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f0be76f8e10> 05/04/2025 18:06:18.445 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/04/2025 18:06:18.445 7f0be7792198 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f0be7630cc0> 05/04/2025 18:06:18.446 7f0be7bf7cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 05/04/2025 18:06:18.446 7f0be7bf7cc0 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED <cfgm_common.tests.test_utils.FakeKombu.Connection object at 0x7f0be7630cc0> 05/04/2025 18:06:22.369 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(376f30c6-562e-4bf6-bf1a-b535c8ebc3f7), feature=l3-gateway 05/04/2025 18:06:22.369 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(376f30c6-562e-4bf6-bf1a-b535c8ebc3f7), feature=vn-interconnect 05/04/2025 18:06:22.370 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=376f30c6-562e-4bf6-bf1a-b535c8ebc3f7, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:22.412 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(376f30c6-562e-4bf6-bf1a-b535c8ebc3f7), feature=l3-gateway 05/04/2025 18:06:22.412 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_in_network(376f30c6-562e-4bf6-bf1a-b535c8ebc3f7), feature=vn-interconnect 05/04/2025 18:06:22.412 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=376f30c6-562e-4bf6-bf1a-b535c8ebc3f7, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:22.721 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:22 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_4921b513-3475-44f3-a09f-904daebd95a4__', '__contrail_lr_internal_vn_4921b513-3475-44f3-a09f-904daebd95a4__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_4921b513-3475-44f3-a09f-904daebd95a4__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_4921b513-3475-44f3-a09f-904daebd95a4__', '__contrail_lr_internal_vn_4921b513-3475-44f3-a09f-904daebd95a4__'], 'id_perms': {'created': '2025-05-05T01:06:22.696149', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-05-05T01:06:22.696149', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9287691235499798220, 'uuid_mslong': 17056645874748573414}}, 'parent_type': 'virtual-network', 'parent_uuid': 'ad0517bd-c000-4db8-b5c1-2afca0697def', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'ecb5610d-9805-46e6-80e4-81f86b796ecc'}, 'oper': 'CREATE', 'request-id': 'req-8f1b9e64-d532-4854-b4d4-c4b97841add4', 'type': 'routing_instance', 'uuid': 'ecb5610d-9805-46e6-80e4-81f86b796ecc'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:22.799 7f0be75deeb8 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'4921b513-3475-44f3-a09f-904daebd95a4'}. This may indicate that the cluster needs a 'nodetool repair'. 05/04/2025 18:06:22.799 7f0be75deeb8 [contrail-device-manager] [WARNING]: logical_router uuid 4921b513-3475-44f3-a09f-904daebd95a4 update caused NoIdError 05/04/2025 18:06:24.998 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(37444d18-03f9-4a83-b70f-e7abdeea754f), feature=l3-gateway 05/04/2025 18:06:24.998 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(37444d18-03f9-4a83-b70f-e7abdeea754f), feature=vn-interconnect 05/04/2025 18:06:24.998 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=37444d18-03f9-4a83-b70f-e7abdeea754f, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:25.040 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(37444d18-03f9-4a83-b70f-e7abdeea754f), feature=l3-gateway 05/04/2025 18:06:25.040 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_inet0(37444d18-03f9-4a83-b70f-e7abdeea754f), feature=vn-interconnect 05/04/2025 18:06:25.040 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=37444d18-03f9-4a83-b70f-e7abdeea754f, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:25.350 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:25 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_8272d9f2-1344-4b7f-9acd-64209bdcc3ca__', '__contrail_lr_internal_vn_8272d9f2-1344-4b7f-9acd-64209bdcc3ca__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_8272d9f2-1344-4b7f-9acd-64209bdcc3ca__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_8272d9f2-1344-4b7f-9acd-64209bdcc3ca__', '__contrail_lr_internal_vn_8272d9f2-1344-4b7f-9acd-64209bdcc3ca__'], 'id_perms': {'created': '2025-05-05T01:06:25.326002', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-05-05T01:06:25.326002', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13125864130707517690, 'uuid_mslong': 7254672290472478255}}, 'parent_type': 'virtual-network', 'parent_uuid': 'b27a8982-dc28-4aca-9e72-b49e9669cb78', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '64adc602-e37b-462f-b628-6fe210dcf0fa'}, 'oper': 'CREATE', 'request-id': 'req-5801ec09-6172-4dae-9df8-39c34f059240', 'type': 'routing_instance', 'uuid': '64adc602-e37b-462f-b628-6fe210dcf0fa'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:25.429 7f0be75deeb8 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'8272d9f2-1344-4b7f-9acd-64209bdcc3ca'}. This may indicate that the cluster needs a 'nodetool repair'. 05/04/2025 18:06:25.429 7f0be75deeb8 [contrail-device-manager] [WARNING]: logical_router uuid 8272d9f2-1344-4b7f-9acd-64209bdcc3ca update caused NoIdError 05/04/2025 18:06:27.614 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(b8b4617c-79da-4276-912d-17e98cd08a70), feature=vn-interconnect 05/04/2025 18:06:27.614 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(b8b4617c-79da-4276-912d-17e98cd08a70), feature=l3-gateway 05/04/2025 18:06:27.614 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=b8b4617c-79da-4276-912d-17e98cd08a70, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:27.658 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(b8b4617c-79da-4276-912d-17e98cd08a70), feature=vn-interconnect 05/04/2025 18:06:27.658 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_one_vn_two_dhcp_servers(b8b4617c-79da-4276-912d-17e98cd08a70), feature=l3-gateway 05/04/2025 18:06:27.658 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=b8b4617c-79da-4276-912d-17e98cd08a70, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:27.967 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:27 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_796e65b5-1cae-4f7f-86d2-5f97ca21fccc__', '__contrail_lr_internal_vn_796e65b5-1cae-4f7f-86d2-5f97ca21fccc__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_796e65b5-1cae-4f7f-86d2-5f97ca21fccc__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_796e65b5-1cae-4f7f-86d2-5f97ca21fccc__', '__contrail_lr_internal_vn_796e65b5-1cae-4f7f-86d2-5f97ca21fccc__'], 'id_perms': {'created': '2025-05-05T01:06:27.941476', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-05-05T01:06:27.941476', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11294437090127189810, 'uuid_mslong': 9216125129939897391}}, 'parent_type': 'virtual-network', 'parent_uuid': '29986f9f-7979-42b1-b077-b605ad1382df', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '7fe640f9-ff9e-4c2f-9cbd-e6b167071732'}, 'oper': 'CREATE', 'request-id': 'req-13dbdcc7-6f34-45e5-87c6-8762c24b2cba', 'type': 'routing_instance', 'uuid': '7fe640f9-ff9e-4c2f-9cbd-e6b167071732'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:28.046 7f0be75deeb8 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'796e65b5-1cae-4f7f-86d2-5f97ca21fccc'}. This may indicate that the cluster needs a 'nodetool repair'. 05/04/2025 18:06:28.046 7f0be75deeb8 [contrail-device-manager] [WARNING]: logical_router uuid 796e65b5-1cae-4f7f-86d2-5f97ca21fccc update caused NoIdError 05/04/2025 18:06:30.242 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(2515e903-ce1e-40db-a3b2-91b6717a3e43), feature=vn-interconnect 05/04/2025 18:06:30.242 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(2515e903-ce1e-40db-a3b2-91b6717a3e43), feature=l3-gateway 05/04/2025 18:06:30.243 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=2515e903-ce1e-40db-a3b2-91b6717a3e43, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:30.285 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(2515e903-ce1e-40db-a3b2-91b6717a3e43), feature=vn-interconnect 05/04/2025 18:06:30.286 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_lrs_with_dhcp_in_one(2515e903-ce1e-40db-a3b2-91b6717a3e43), feature=l3-gateway 05/04/2025 18:06:30.286 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=2515e903-ce1e-40db-a3b2-91b6717a3e43, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:30.592 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:30 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_775dfca1-1855-4379-9525-6728dcf700eb__', '__contrail_lr_internal_vn_775dfca1-1855-4379-9525-6728dcf700eb__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_775dfca1-1855-4379-9525-6728dcf700eb__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_775dfca1-1855-4379-9525-6728dcf700eb__', '__contrail_lr_internal_vn_775dfca1-1855-4379-9525-6728dcf700eb__'], 'id_perms': {'created': '2025-05-05T01:06:30.569068', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-05-05T01:06:30.569068', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12512467442747454875, 'uuid_mslong': 8338606528626445070}}, 'parent_type': 'virtual-network', 'parent_uuid': '00ef8120-f466-4a96-8784-1dc795c828ba', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '73b8ae75-3c88-4b0e-ada5-36d8649a359b'}, 'oper': 'CREATE', 'request-id': 'req-e6c0dd85-c889-4b04-993d-fa8ed3f5977a', 'type': 'routing_instance', 'uuid': '73b8ae75-3c88-4b0e-ada5-36d8649a359b'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:30.883 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:30 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_7e6adeef-ef80-4ad2-a05b-a819e4effbe2__', '__contrail_lr_internal_vn_7e6adeef-ef80-4ad2-a05b-a819e4effbe2__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_7e6adeef-ef80-4ad2-a05b-a819e4effbe2__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_7e6adeef-ef80-4ad2-a05b-a819e4effbe2__', '__contrail_lr_internal_vn_7e6adeef-ef80-4ad2-a05b-a819e4effbe2__'], 'id_perms': {'created': '2025-05-05T01:06:30.861011', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-05-05T01:06:30.861011', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12897168005832309905, 'uuid_mslong': 562626337779895776}}, 'parent_type': 'virtual-network', 'parent_uuid': '46ad3e2f-bb8b-4ba0-a0a1-9b8811365ac0', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '07ced9ac-5ca6-4de0-b2fb-f1f81c2afc91'}, 'oper': 'CREATE', 'request-id': 'req-8c93beeb-baa3-4b7b-9189-138bd921b762', 'type': 'routing_instance', 'uuid': '07ced9ac-5ca6-4de0-b2fb-f1f81c2afc91'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:30.961 7f0be75deeb8 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'775dfca1-1855-4379-9525-6728dcf700eb'}. This may indicate that the cluster needs a 'nodetool repair'. 05/04/2025 18:06:30.961 7f0be75deeb8 [contrail-device-manager] [WARNING]: logical_router uuid 775dfca1-1855-4379-9525-6728dcf700eb update caused NoIdError 05/04/2025 18:06:31.015 7f0be75deeb8 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'7e6adeef-ef80-4ad2-a05b-a819e4effbe2'}. This may indicate that the cluster needs a 'nodetool repair'. 05/04/2025 18:06:31.015 7f0be75deeb8 [contrail-device-manager] [WARNING]: logical_router uuid 7e6adeef-ef80-4ad2-a05b-a819e4effbe2 update caused NoIdError 05/04/2025 18:06:33.322 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(dd60284c-63df-4319-990d-e78bf7ef714a), feature=vn-interconnect 05/04/2025 18:06:33.322 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(dd60284c-63df-4319-990d-e78bf7ef714a), feature=l3-gateway 05/04/2025 18:06:33.322 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=dd60284c-63df-4319-990d-e78bf7ef714a, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:33.365 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(dd60284c-63df-4319-990d-e78bf7ef714a), feature=vn-interconnect 05/04/2025 18:06:33.365 7f0be75deeb8 [contrail-device-manager] [INFO]: Found feature plugin for pr=routertest.test_dm_ansible_dhcp_relay.TestAnsibleDhcpRelayDM.test_dhcp_relay_config_push_two_vns_in_network(dd60284c-63df-4319-990d-e78bf7ef714a), feature=l3-gateway 05/04/2025 18:06:33.366 7f0be75deeb8 [contrail-device-manager] [INFO]: Found ansible plugin pr=dd60284c-63df-4319-990d-e78bf7ef714a, role/vendor/product=spine:juniper:qfx10008 05/04/2025 18:06:33.908 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:33 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_5893004b-1b43-4dad-b9a9-8d69438d144e__', '__contrail_lr_internal_vn_5893004b-1b43-4dad-b9a9-8d69438d144e__'], 'obj_dict': {'display_name': '__contrail_lr_internal_vn_5893004b-1b43-4dad-b9a9-8d69438d144e__', 'fq_name': ['default-domain', 'default-project', '__contrail_lr_internal_vn_5893004b-1b43-4dad-b9a9-8d69438d144e__', '__contrail_lr_internal_vn_5893004b-1b43-4dad-b9a9-8d69438d144e__'], 'id_perms': {'created': '2025-05-05T01:06:33.882859', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-05-05T01:06:33.882859', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13644264500021642834, 'uuid_mslong': 17607262901794522607}}, 'parent_type': 'virtual-network', 'parent_uuid': '4a619de4-92a9-4d3b-839a-620c4ba2cc1f', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'f4599052-6860-45ef-bd5a-2a45b9434652'}, 'oper': 'CREATE', 'request-id': 'req-d4af1bfb-5dea-4bfd-93d1-a2624d978c71', 'type': 'routing_instance', 'uuid': 'f4599052-6860-45ef-bd5a-2a45b9434652'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:33.913 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:33 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', 'lr-test.test_dm_ansible_dhcp_relay.TestAnsibleDh...DM.test_dhcp_relay_config_push_two_vns_in_network'], 'oper': 'UPDATE', 'request-id': 'req-d4af1bfb-5dea-4bfd-93d1-a2624d978c71', 'type': 'logical_router', 'uuid': '5893004b-1b43-4dad-b9a9-8d69438d144e'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:33.924 7f0be75deeb8 [contrail-device-manager] [ERROR]: AttributeError Python 3.6.8: /root/work/build/debug/config/device-manager/.tox/py36/bin/python Sun May 4 18:06:33 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in _vnc_subscribe_callback(self=<device_manager.dm_amqp.DMAmqpHandle object>, oper_info={'fq_name': ['default-domain', 'default-project', 'lr-test.test_dm_ansible_dhcp_relay.TestAnsibleDh...DM.test_dhcp_relay_config_push_two_vns_in_network'], 'oper': 'UPDATE', 'request-id': 'req-34b5a9c6-cde5-42cf-ad94-fd8df0a111d7', 'type': 'logical_router', 'uuid': '5893004b-1b43-4dad-b9a9-8d69438d144e'}) 82 try: 83 self.oper_info = oper_info 84 self.vnc_subscribe_actions() 85 86 except ConnectionError: self = <device_manager.dm_amqp.DMAmqpHandle object> self.vnc_subscribe_actions = <bound method VncAmqpHandle.vnc_subscribe_actions of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py in vnc_subscribe_actions(self=<device_manager.dm_amqp.DMAmqpHandle object>) 146 (self. obj_type, obj_id, oper)) 147 return 148 self.evaluate_dependency() 149 150 def _get_key_from_oper_info(self): self = <device_manager.dm_amqp.DMAmqpHandle object> self.evaluate_dependency = <bound method DMAmqpHandle.evaluate_dependency of <device_manager.dm_amqp.DMAmqpHandle object>> /root/work/build/debug/config/device-manager/device_manager/dm_amqp.py in evaluate_dependency(self=<device_manager.dm_amqp.DMAmqpHandle object>) 49 []): 50 pr = PhysicalRouterDM.get(pr_id) 51 if pr is not None: 52 pr.set_config_state() 53 pr.uve_send() pr = <device_manager.db.PhysicalRouterDM object> pr.set_config_state = <bound method PhysicalRouterDM.set_config_state of <device_manager.db.PhysicalRouterDM object>> /root/work/build/debug/config/device-manager/device_manager/db.py in set_config_state(self=<device_manager.db.PhysicalRouterDM object>, reset_retry_count=True) 1044 self.nc_q.put_nowait(1) 1045 except queue.Full: 1046 self._logger.warn(f"queue is full") 1047 1048 # end self = <device_manager.db.PhysicalRouterDM object> self._logger = <device_manager.logger.DeviceManagerLogger object> self._logger.warn undefined AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' __cause__ = None __class__ = <class 'AttributeError'> __context__ = Full() __delattr__ = <method-wrapper '__delattr__' of AttributeError object> __dict__ = {} __dir__ = <built-in method __dir__ of AttributeError object> __doc__ = 'Attribute not found.' __eq__ = <method-wrapper '__eq__' of AttributeError object> __format__ = <built-in method __format__ of AttributeError object> __ge__ = <method-wrapper '__ge__' of AttributeError object> __getattribute__ = <method-wrapper '__getattribute__' of AttributeError object> __gt__ = <method-wrapper '__gt__' of AttributeError object> __hash__ = <method-wrapper '__hash__' of AttributeError object> __init__ = <method-wrapper '__init__' of AttributeError object> __init_subclass__ = <built-in method __init_subclass__ of type object> __le__ = <method-wrapper '__le__' of AttributeError object> __lt__ = <method-wrapper '__lt__' of AttributeError object> __ne__ = <method-wrapper '__ne__' of AttributeError object> __new__ = <built-in method __new__ of type object> __reduce__ = <built-in method __reduce__ of AttributeError object> __reduce_ex__ = <built-in method __reduce_ex__ of AttributeError object> __repr__ = <method-wrapper '__repr__' of AttributeError object> __setattr__ = <method-wrapper '__setattr__' of AttributeError object> __setstate__ = <built-in method __setstate__ of AttributeError object> __sizeof__ = <built-in method __sizeof__ of AttributeError object> __str__ = <method-wrapper '__str__' of AttributeError object> __subclasshook__ = <built-in method __subclasshook__ of type object> __suppress_context__ = False __traceback__ = <traceback object> args = ("'DeviceManagerLogger' object has no attribute 'warn'",) with_traceback = <built-in method with_traceback of AttributeError object> The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1044, in set_config_state self.nc_q.put_nowait(1) File "src/gevent/queue.py", line 290, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 296, in gevent._gevent_cqueue.Queue.put_nowait File "src/gevent/queue.py", line 288, in gevent._gevent_cqueue.Queue.put queue.Full During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 84, in _vnc_subscribe_callback self.vnc_subscribe_actions() File "/root/work/build/debug/config/device-manager/.tox/py36/lib/python3.6/site-packages/cfgm_common/vnc_amqp.py", line 148, in vnc_subscribe_actions self.evaluate_dependency() File "/root/work/build/debug/config/device-manager/device_manager/dm_amqp.py", line 52, in evaluate_dependency pr.set_config_state() File "/root/work/build/debug/config/device-manager/device_manager/db.py", line 1046, in set_config_state self._logger.warn(f"queue is full") AttributeError: 'DeviceManagerLogger' object has no attribute 'warn' 05/04/2025 18:06:34.946 7f0be75deeb8 [contrail-device-manager] [WARNING]: Inconsistency discovered. wanted=1, received=0, missing={'5893004b-1b43-4dad-b9a9-8d69438d144e'}. This may indicate that the cluster needs a 'nodetool repair'. 05/04/2025 18:06:34.946 7f0be75deeb8 [contrail-device-manager] [WARNING]: logical_router uuid 5893004b-1b43-4dad-b9a9-8d69438d144e update caused NoIdError 05/04/2025 18:06:39.502 7f0be7ab8048 [contrail-device-manager] [INFO]: SANDESH: CONNECT TO COLLECTOR: False 05/04/2025 18:06:39.513 7f0be7ab8048 [contrail-device-manager] [INFO]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_INFO] 05/04/2025 18:06:39.513 7f0be7ab8048 [contrail-device-manager] [INFO]: SANDESH: Logging: FILE: [None] -> [device_manager_TestAnsibleDciIntraFabric.log]