2025-10-09 13:40:06,095 - DEBUG - [10.0.0.41]: Running cmd : hostname 2025-10-09 13:40:06,336 - DEBUG - Output : cn-jenkins-deploy-platform-juju-os-588-1 2025-10-09 13:40:06,336 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2025-10-09 13:40:06,428 - DEBUG - Output : cn-jenkins-deploy-platform-juju-os-588-1 2025-10-09 13:40:06,428 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-09 13:40:06,552 - DEBUG - Output : NAMES contrail_test_NPiHvce3J vrouter-vrouter-agent-1 vrouter-nodemgr-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 analytics_database-cassandra-1 analytics_snmp-snmp-collector-1 analytics_snmp-nodemgr-1 analytics_snmp-topology-1 analytics_alarm-kafka-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics-api-1 analytics-collector-1 analytics-nodemgr-1 redis-redis-1 redis-stunnel-1 webui-web-1 webui-job-1 control-named-1 control-dns-1 control-control-1 control-nodemgr-1 config_database-rabbitmq-1 config_database-nodemgr-1 config_database-cassandra-1 config_database-zookeeper-1 config_api-devicemgr-1 config_api-nodemgr-1 config_api-dnsmasq-1 config_api-svcmonitor-1 config_api-schema-1 config_api-api-1 2025-10-09 13:40:06,552 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.0.0.41 | head -n 1 | awk '{print $2}' 2025-10-09 13:40:06,655 - DEBUG - Output : cn-jenkins-deploy-platform-juju-os-588-1 2025-10-09 13:40:06,655 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-09 13:40:06,766 - DEBUG - Output : 127.0.0.1/8 252.41.0.1/8 10.233.130.1/24 172.17.0.1/16 10.0.0.41/24 2025-10-09 13:40:06,766 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-09 13:40:06,870 - DEBUG - Output : 10.0.0.41/24 2025-10-09 13:40:18,302 - INFO - Using existing project ['admin_domain', 'admin'](2f0ea955-6c00-46c1-8ddd-92c9167f91d0) 2025-10-09 13:40:18,474 - INFO - ================================================================================ 2025-10-09 13:40:18,474 - INFO - STARTING TEST : test_contrail_status 2025-10-09 13:40:18,474 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2025-10-09 13:40:18,714 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:40:18,768 - DEBUG - Nothing to compare xmpp stats {'10.0.0.41': {'10.0.0.41': '0'}} with 2025-10-09 13:40:18,768 - INFO - Initial checks done. Running the testcase now 2025-10-09 13:40:18,768 - INFO - 2025-10-09 13:40:18,769 - DEBUG - [10.0.0.41]: Running cmd : docker ps -f status=running --format {{.Names}} 2>/dev/null 2025-10-09 13:40:18,854 - DEBUG - Output : contrail_test_NPiHvce3J vrouter-vrouter-agent-1 vrouter-nodemgr-1 vrouter-provisioner-1 control-provisioner-1 config_api-provisioner-1 analytics_database-provisioner-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 analytics_database-cassandra-1 analytics_snmp-provisioner-1 analytics_snmp-snmp-collector-1 analytics_snmp-nodemgr-1 analytics_snmp-topology-1 analytics_alarm-kafka-1 analytics_alarm-provisioner-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics-api-1 analytics-collector-1 analytics-nodemgr-1 analytics-provisioner-1 redis-redis-1 redis-stunnel-1 webui-web-1 webui-job-1 control-named-1 control-dns-1 control-control-1 control-nodemgr-1 config_database-rabbitmq-1 config_database-nodemgr-1 config_database-provisioner-1 config_database-cassandra-1 config_database-zookeeper-1 config_api-devicemgr-1 config_api-nodemgr-1 config_api-dnsmasq-1 config_api-svcmonitor-1 config_api-schema-1 config_api-api-1 2025-10-09 13:40:18,854 - INFO - 10.0.0.41 2025-10-09 13:40:18,854 - INFO - config 2025-10-09 13:40:18,912 - INFO - config-nodemgr:active 2025-10-09 13:40:18,967 - INFO - api-server:active 2025-10-09 13:40:19,023 - INFO - schema:active 2025-10-09 13:40:19,079 - INFO - svc-monitor:active 2025-10-09 13:40:19,135 - INFO - device-manager:active 2025-10-09 13:40:19,135 - INFO - config-database 2025-10-09 13:40:19,135 - INFO - config-cassandra:active 2025-10-09 13:40:19,135 - INFO - config-zookeeper:active 2025-10-09 13:40:19,135 - INFO - config-rabbitmq:active 2025-10-09 13:40:19,135 - INFO - vrouter 2025-10-09 13:40:19,191 - INFO - vrouter-nodemgr:active 2025-10-09 13:40:19,247 - INFO - agent:active 2025-10-09 13:40:19,247 - INFO - control 2025-10-09 13:40:19,303 - INFO - control-nodemgr:active 2025-10-09 13:40:19,355 - INFO - control:active 2025-10-09 13:40:19,355 - INFO - named:active 2025-10-09 13:40:19,411 - INFO - dns:active 2025-10-09 13:40:19,411 - INFO - analytics 2025-10-09 13:40:19,467 - INFO - analytics-nodemgr:active 2025-10-09 13:40:19,518 - INFO - analytics-api:active 2025-10-09 13:40:19,575 - INFO - collector:active 2025-10-09 13:40:19,575 - INFO - analytics-database 2025-10-09 13:40:19,575 - INFO - analytics-cassandra:active 2025-10-09 13:40:19,635 - INFO - analyticsdb-nodemgr:active 2025-10-09 13:40:19,691 - INFO - query-engine:active 2025-10-09 13:40:19,691 - INFO - webui 2025-10-09 13:40:19,691 - INFO - webui:active 2025-10-09 13:40:19,691 - INFO - webui-middleware:active 2025-10-09 13:40:19,691 - INFO - redis:active 2025-10-09 13:40:19,691 - INFO - analytics_snmp 2025-10-09 13:40:19,747 - INFO - snmp-collector:active 2025-10-09 13:40:19,747 - INFO - snmp-topology:active 2025-10-09 13:40:19,747 - INFO - analytics_alarm 2025-10-09 13:40:19,747 - INFO - alarmgen:active 2025-10-09 13:40:19,747 - INFO - All the contrail services are up on all nodes 2025-10-09 13:40:19,747 - INFO - contrail-status passed 2025-10-09 13:40:19,958 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:40:20,008 - DEBUG - No XMPP flaps were noticed during the test
2025-10-09 13:40:20,008 - INFO - END TEST : test_contrail_status : PASSED[0:00:02]
2025-10-09 13:40:20,008 - INFO - -------------------------------------------------------------------------------- 2025-10-09 13:40:30,673 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:40:35,079 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-91867243:ctest-vn22-36940797:ctest-vn22-36940797 2025-10-09 13:40:35,131 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-91867243:ctest-vn22-36940797:ctest-vn22-36940797 2025-10-09 13:40:35,179 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-91867243:ctest-vn22-36940797:ctest-vn22-36940797 2025-10-09 13:40:57,911 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-91867243:ctest-vn22-36940797 2025-10-09 13:40:57,967 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-91867243:ctest-vn22-36940797:ctest-vn22-36940797 2025-10-09 13:40:58,019 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-AnalyticsBasicTestSanity-91867243:ctest-vn22-36940797:ctest-vn22-36940797 2025-10-09 13:40:58,390 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:00,036 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:00,088 - DEBUG - [10.0.0.41]: Running cmd : docker exec --privileged -it config_api-api-1 /bin/bash -c 'db-manage check' 2025-10-09 13:41:03,218 - DEBUG - Output : 2025-10-09 13:41:03,037 INFO: (v1.41) Checker check_zk_mode_and_node_count: Success 2025-10-09 13:41:03,091 ERROR: (v1.41) Checker check_cassandra_keyspace_replication: Exception, NoHostAvailable Python 3.9.21: /usr/bin/python3 Thu Oct 9 13:41:03 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in wrapper(*args=(,), **kwargs={}) 1535 self = args[0] 1536 try: 1537 errors = func(*args, **kwargs) 1538 if not errors: 1539 self._logger.info('(v%s) Checker %s: Success', __version__, errors undefined func = args = (,) kwargs = {} /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in check_cassandra_keyspace_replication(self=) 1635 raise CassandraConnectionError("error, {}: {}".format( 1636 error, utils.detailed_traceback())) 1637 temp_cluster.connect() 1638 for ks_name, _ in self._db_info: 1639 if self._api_args.cluster_id: temp_cluster = temp_cluster.connect = > /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection._reconnect_internal() NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) errors = {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1537, in wrapper errors = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1637, in check_cassandra_keyspace_replication temp_cluster.connect() File "cassandra/cluster.py", line 1717, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1753, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1740, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 3543, in cassandra.cluster.ControlConnection.connect File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._reconnect_internal cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1537, in wrapper errors = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1637, in check_cassandra_keyspace_replication temp_cluster.connect() File "cassandra/cluster.py", line 1717, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1753, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1740, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 3543, in cassandra.cluster.ControlConnection.connect File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._reconnect_internal cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) NoHostAvailable Python 3.9.21: /usr/bin/python3 Thu Oct 9 13:41:03 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/local/bin/db-manage in 3 from vnc_cfg_api_server.db_manage import main 4 if __name__ == '__main__': 5 if sys.argv[0].endswith('.exe'): 6 sys.argv[0] = sys.argv[0][:-4] 7 sys.exit(main()) sys = sys.exit = main = /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in main() 3451 verb = args.operation 3452 if 'db_%s' % (verb) in globals(): 3453 return globals()['db_%s' % (verb)](args, api_args) 3454 3455 if getattr(DatabaseChecker, verb, None): builtinglobals = verb = 'check' args = Namespace(operation='check', api_conf='/etc/cont...1024, log_file='/var/log/contrail/db_manage.log') api_args = Namespace(conf_file=['/etc/contrail/contrail-api...esh_base.SandeshConfig object at 0x7f0213c9a0d0>) /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in db_check(args=Namespace(operation='check', api_conf='/etc/cont...1024, log_file='/var/log/contrail/db_manage.log'), api_args=Namespace(conf_file=['/etc/contrail/contrail-api...esh_base.SandeshConfig object at 0x7f0213c9a0d0>)) 3358 # Mode and node count check across all nodes 3359 db_checker.check_zk_mode_and_node_count() 3360 db_checker.check_cassandra_keyspace_replication() 3361 # Obj UUID cassandra inconsistencies 3362 db_checker.check_obj_mandatory_fields() db_checker = db_checker.check_cassandra_keyspace_replication = > /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in wrapper(*args=(,), **kwargs={}) 1535 self = args[0] 1536 try: 1537 errors = func(*args, **kwargs) 1538 if not errors: 1539 self._logger.info('(v%s) Checker %s: Success', __version__, errors undefined func = args = (,) kwargs = {} /usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py in check_cassandra_keyspace_replication(self=) 1635 raise CassandraConnectionError("error, {}: {}".format( 1636 error, utils.detailed_traceback())) 1637 temp_cluster.connect() 1638 for ks_name, _ in self._db_info: 1639 if self._api_args.cluster_id: temp_cluster = temp_cluster.connect = > /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.Cluster.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection.connect() /cassandra/cluster.py in cassandra.cluster.ControlConnection._reconnect_internal() NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {'errors': {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}} __dir__ = __doc__ = '\n Raised when an operation is attempted but a...ct, closed, or resulted in errors when used.\n ' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __module__ = 'cassandra.cluster' __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = __weakref__ = None args = ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) errors = {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')} with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/local/bin/db-manage", line 7, in sys.exit(main()) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 3453, in main return globals()['db_%s' % (verb)](args, api_args) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 3360, in db_check db_checker.check_cassandra_keyspace_replication() File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1537, in wrapper errors = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/vnc_cfg_api_server/db_manage.py", line 1637, in check_cassandra_keyspace_replication temp_cluster.connect() File "cassandra/cluster.py", line 1717, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1753, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 1740, in cassandra.cluster.Cluster.connect File "cassandra/cluster.py", line 3543, in cassandra.cluster.ControlConnection.connect File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._reconnect_internal cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.0.41:9041': ConnectionResetError(104, 'Connection reset by peer')}) 2025-10-09 13:41:03,476 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:03,827 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:04,125 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:04,398 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:04,670 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:08,355 - INFO - admin user already present 2025-10-09 13:41:09,480 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:13,375 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:13,427 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:13,475 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:15,739 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663:ctest-vn1-74370663 2025-10-09 13:41:15,787 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663:ctest-vn1-74370663 2025-10-09 13:41:15,839 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663:ctest-vn1-74370663 2025-10-09 13:41:27,256 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:62e5af47-0c58-487b-917c-03680f148f9d 2025-10-09 13:41:27,311 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663:ctest-vn1-74370663 2025-10-09 13:41:27,363 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:62e5af47-0c58-487b-917c-03680f148f9d 2025-10-09 13:41:39,264 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:93099b23-4998-456d-a1da-8780ef74de0e 2025-10-09 13:41:39,315 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:39,371 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:93099b23-4998-456d-a1da-8780ef74de0e 2025-10-09 13:41:41,156 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:62e5af47-0c58-487b-917c-03680f148f9d 2025-10-09 13:41:41,208 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095 2025-10-09 13:41:41,262 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:41,311 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=3 2025-10-09 13:41:47,872 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663 2025-10-09 13:41:47,923 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663:ctest-vn1-74370663 2025-10-09 13:41:47,975 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-vn1-74370663:ctest-vn1-74370663 2025-10-09 13:41:48,568 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095 2025-10-09 13:41:48,619 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:48,667 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-FloatingipBasicTestSanity-54952622:ctest-fvn-45717095:ctest-fvn-45717095 2025-10-09 13:41:49,165 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:53,213 - INFO - ctest-TestHeat-30984721 user already added to project 2025-10-09 13:41:53,569 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:41:54,600 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:03,895 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:12,873 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:17,233 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:20,355 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:20,775 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:30,104 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8c332280-d2eb-4068-b253-a0ebf2f8debf 2025-10-09 13:42:30,156 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestPorts-56304314:ctest-vn1-94323002:ctest-vn1-94323002 2025-10-09 13:42:30,207 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8c332280-d2eb-4068-b253-a0ebf2f8debf 2025-10-09 13:42:46,113 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:99677ccb-6d04-4c9c-997b-d742662c2010 2025-10-09 13:42:46,164 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestPorts-56304314:ctest-vn1-94323002:ctest-vn1-94323002 2025-10-09 13:42:46,215 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:99677ccb-6d04-4c9c-997b-d742662c2010 2025-10-09 13:42:53,223 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:42:57,554 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:43:09,632 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1c49831d-4681-4be7-844e-1a4bf5cae2ad 2025-10-09 13:43:09,684 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRouters-99796359:ctest-vn1-79123492:ctest-vn1-79123492 2025-10-09 13:43:09,735 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1c49831d-4681-4be7-844e-1a4bf5cae2ad 2025-10-09 13:43:21,596 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e43e4d9e-72e3-4fb4-bcb5-dc69047bf6f0 2025-10-09 13:43:21,651 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRouters-99796359:ctest-vn2-28060195:ctest-vn2-28060195 2025-10-09 13:43:21,703 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:e43e4d9e-72e3-4fb4-bcb5-dc69047bf6f0 2025-10-09 13:43:45,125 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:43:49,353 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:02,576 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cdf43f08-e0ff-4e19-b929-82cbe370e24f 2025-10-09 13:44:02,628 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRoutersBasic-15726802:ctest-vn-16762792:ctest-vn-16762792 2025-10-09 13:44:02,683 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cdf43f08-e0ff-4e19-b929-82cbe370e24f 2025-10-09 13:44:14,353 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:dc2cc7c9-a8c1-4ffb-b09b-7a9ab836555d 2025-10-09 13:44:14,403 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRoutersBasic-15726802:ctest-ext_vn-75057742:ctest-ext_vn-75057742 2025-10-09 13:44:14,455 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:dc2cc7c9-a8c1-4ffb-b09b-7a9ab836555d 2025-10-09 13:44:24,273 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:28,809 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:35,544 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-03044196:ctest-vn-03044196 2025-10-09 13:44:35,595 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-03044196:ctest-vn-03044196 2025-10-09 13:44:35,643 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-03044196:ctest-vn-03044196 2025-10-09 13:44:36,990 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-50941452:ctest-vn-50941452 2025-10-09 13:44:37,044 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-50941452:ctest-vn-50941452 2025-10-09 13:44:37,095 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-50941452:ctest-vn-50941452 2025-10-09 13:44:38,340 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-50941452 2025-10-09 13:44:38,393 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-50941452:ctest-vn-50941452 2025-10-09 13:44:38,443 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-50941452:ctest-vn-50941452 2025-10-09 13:44:39,040 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-03044196 2025-10-09 13:44:39,091 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-03044196:ctest-vn-03044196 2025-10-09 13:44:39,139 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyIpv4v6-96829513:ctest-vn-03044196:ctest-vn-03044196 2025-10-09 13:44:39,637 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:43,836 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:44,243 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:49,135 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-11752473:ctest-vn4-75533263:ctest-vn4-75533263 2025-10-09 13:44:49,187 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-11752473:ctest-vn4-75533263:ctest-vn4-75533263 2025-10-09 13:44:49,243 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-11752473:ctest-vn4-75533263:ctest-vn4-75533263 2025-10-09 13:44:50,045 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-11752473:ctest-vn4-75533263 2025-10-09 13:44:50,095 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-11752473:ctest-vn4-75533263:ctest-vn4-75533263 2025-10-09 13:44:50,147 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicyNegativeIpv4v6-11752473:ctest-vn4-75533263:ctest-vn4-75533263 2025-10-09 13:44:50,865 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:51,264 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:44:57,760 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:45:05,739 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518:ctest-vn-31196518 2025-10-09 13:45:05,796 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518:ctest-vn-31196518 2025-10-09 13:45:05,847 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518:ctest-vn-31196518 2025-10-09 13:45:07,288 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562:ctest-vn-05796562 2025-10-09 13:45:07,339 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562:ctest-vn-05796562 2025-10-09 13:45:07,395 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562:ctest-vn-05796562 2025-10-09 13:45:16,752 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:993dc8bd-e9c0-47ee-9623-0e17ab440b4c 2025-10-09 13:45:16,804 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518:ctest-vn-31196518 2025-10-09 13:45:16,856 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:993dc8bd-e9c0-47ee-9623-0e17ab440b4c 2025-10-09 13:45:30,604 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1215b7d5-5fd7-4dfd-804c-786f83845344 2025-10-09 13:45:30,655 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562:ctest-vn-05796562 2025-10-09 13:45:30,708 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1215b7d5-5fd7-4dfd-804c-786f83845344 2025-10-09 13:45:43,184 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562 2025-10-09 13:45:43,235 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562:ctest-vn-05796562 2025-10-09 13:45:43,283 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-05796562:ctest-vn-05796562 2025-10-09 13:45:43,871 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518 2025-10-09 13:45:43,923 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518:ctest-vn-31196518 2025-10-09 13:45:43,971 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicPolicy-98788352:ctest-vn-31196518:ctest-vn-31196518 2025-10-09 13:45:44,470 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:45:48,825 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:46:00,932 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5e08df1c-277c-44c7-b4cd-2dc481a77635 2025-10-09 13:46:00,984 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRP-37775500:ctest-bgpaas_vn-51796077:ctest-bgpaas_vn-51796077 2025-10-09 13:46:01,035 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5e08df1c-277c-44c7-b4cd-2dc481a77635 2025-10-09 13:46:09,480 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f6171774-5f65-4e71-b524-0cc68b96472b 2025-10-09 13:46:09,532 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestRP-37775500:ctest-bgpaas_vn-88543367:ctest-bgpaas_vn-88543367 2025-10-09 13:46:09,583 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f6171774-5f65-4e71-b524-0cc68b96472b 2025-10-09 13:46:11,442 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:46:19,001 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:46:23,186 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:46:26,891 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:46:26,943 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:46:26,995 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:46:29,240 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:46:29,291 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:46:29,348 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:46:31,687 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:46:31,735 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:46:31,791 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:46:50,465 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:429095d3-5432-4924-8c00-87838d3ce4fa 2025-10-09 13:46:50,517 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:46:50,567 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b4ba2220-471d-404a-8788-07456ea5d0d1 2025-10-09 13:46:50,619 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:46:50,671 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:40482834-af7a-4852-a0da-c7989ead0488 2025-10-09 13:46:50,723 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:46:50,775 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:40482834-af7a-4852-a0da-c7989ead0488 2025-10-09 13:46:50,828 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:429095d3-5432-4924-8c00-87838d3ce4fa 2025-10-09 13:46:50,880 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b4ba2220-471d-404a-8788-07456ea5d0d1 2025-10-09 13:47:30,112 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:865e156a-f427-4adf-b438-9cf3c4a5a841 2025-10-09 13:47:30,167 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:47:30,215 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:865e156a-f427-4adf-b438-9cf3c4a5a841 2025-10-09 13:47:32,462 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b0939866-b76a-4a18-b292-4e3a70674eb9 2025-10-09 13:47:32,511 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:47:32,563 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b0939866-b76a-4a18-b292-4e3a70674eb9 2025-10-09 13:47:39,680 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:40482834-af7a-4852-a0da-c7989ead0488 2025-10-09 13:47:39,737 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:429095d3-5432-4924-8c00-87838d3ce4fa 2025-10-09 13:47:39,792 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b4ba2220-471d-404a-8788-07456ea5d0d1 2025-10-09 13:47:41,117 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241 2025-10-09 13:47:41,167 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b4ba2220-471d-404a-8788-07456ea5d0d1 2025-10-09 13:47:41,219 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 13:47:41,233 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:47:41,438 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=2 2025-10-09 13:47:41,658 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Layer2RouteReq?x=2 2025-10-09 13:47:41,674 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241 2025-10-09 13:47:41,723 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:47:41,938 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=2 2025-10-09 13:47:42,166 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Layer2RouteReq?x=2 2025-10-09 13:47:42,181 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494 2025-10-09 13:47:42,235 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:429095d3-5432-4924-8c00-87838d3ce4fa 2025-10-09 13:47:42,288 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 13:47:42,300 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:47:42,518 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=4 2025-10-09 13:47:42,730 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Layer2RouteReq?x=4 2025-10-09 13:47:42,744 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494 2025-10-09 13:47:42,795 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:47:43,010 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=4 2025-10-09 13:47:43,222 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Layer2RouteReq?x=4 2025-10-09 13:47:43,238 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237 2025-10-09 13:47:43,287 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:40482834-af7a-4852-a0da-c7989ead0488 2025-10-09 13:47:43,340 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-09 13:47:43,357 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:47:43,566 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=3 2025-10-09 13:47:43,778 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Layer2RouteReq?x=3 2025-10-09 13:47:43,845 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237 2025-10-09 13:47:43,899 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:47:44,110 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=3 2025-10-09 13:47:44,326 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Layer2RouteReq?x=3 2025-10-09 13:47:45,058 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name= 2025-10-09 13:47:45,112 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=5 2025-10-09 13:47:45,196 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b4ba2220-471d-404a-8788-07456ea5d0d1 2025-10-09 13:47:45,358 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name= 2025-10-09 13:47:45,407 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_Inet4UcRouteReq?x=6 2025-10-09 13:47:45,462 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:40482834-af7a-4852-a0da-c7989ead0488 2025-10-09 13:47:54,460 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237 2025-10-09 13:47:54,515 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:47:54,571 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-right_vn-64429237:ctest-right_vn-64429237 2025-10-09 13:47:55,171 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241 2025-10-09 13:47:55,223 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:47:55,271 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-left_vn-38812241:ctest-left_vn-38812241 2025-10-09 13:47:55,891 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494 2025-10-09 13:47:55,943 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:47:55,991 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestSvcRegr-80692013:ctest-mgmt_vn-44271494:ctest-mgmt_vn-44271494 2025-10-09 13:47:56,492 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:48:00,811 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:48:11,176 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:89c521a8-cb71-47e9-93a5-50a67ad6f348 2025-10-09 13:48:11,228 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestvDNSBasic0-49567083:ctest-vn1-vdns-42527408:ctest-vn1-vdns-42527408 2025-10-09 13:48:11,279 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:89c521a8-cb71-47e9-93a5-50a67ad6f348 2025-10-09 13:48:25,597 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_DnsInfo? 2025-10-09 13:48:32,896 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cfc924e6-fecc-48e0-bc35-c350e52f5051 2025-10-09 13:48:32,951 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestvDNSBasic0-49567083:ctest-vn1-vdns-42527408:ctest-vn1-vdns-42527408 2025-10-09 13:48:33,007 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cfc924e6-fecc-48e0-bc35-c350e52f5051 2025-10-09 13:48:47,276 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_DnsInfo? 2025-10-09 13:48:59,725 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:49:03,893 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:49:14,192 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:dc6bc379-f518-454b-af7a-c943f7802a9c 2025-10-09 13:49:14,244 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVNx-11419170:ctest-vn-97001589:ctest-vn-97001589 2025-10-09 13:49:14,299 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:dc6bc379-f518-454b-af7a-c943f7802a9c 2025-10-09 13:49:25,948 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b53f608b-cf77-4229-ad7e-e37e532da5b5 2025-10-09 13:49:25,999 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVNx-11419170:ctest-vn-97001589:ctest-vn-97001589 2025-10-09 13:49:26,051 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:b53f608b-cf77-4229-ad7e-e37e532da5b5 2025-10-09 13:49:35,521 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:49:39,841 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:49:48,185 - DEBUG - [10.0.0.41]: Running cmd : docker exec --privileged -it vrouter-vrouter-agent-1 /bin/bash -c 'echo "10.0.0.41 cn-jenkins-deploy-platform-juju-os-588-1-test" >> /etc/hosts' 2025-10-09 13:49:48,334 - DEBUG - Output : 2025-10-09 13:49:49,196 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:0066bd5e-a842-42f6-ad93-8816e632b054 2025-10-09 13:49:49,248 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn2_metadata-00410551:ctest-vn2_metadata-00410551 2025-10-09 13:49:49,299 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:0066bd5e-a842-42f6-ad93-8816e632b054 2025-10-09 13:49:58,860 - DEBUG - [10.0.0.41]: Running cmd : docker exec --privileged -it vrouter-vrouter-agent-1 /bin/bash -c 'sed -i '$ d' /etc/hosts' 2025-10-09 13:49:59,034 - DEBUG - Output : sed: -e expression #1, char 1: missing command 2025-10-09 13:50:01,940 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:50:02,342 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:50:10,692 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:748bcd36-7fe8-4190-8ab2-8fdfa33e8f7e 2025-10-09 13:50:10,765 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-56687186:ctest-vn-56687186 2025-10-09 13:50:10,823 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:748bcd36-7fe8-4190-8ab2-8fdfa33e8f7e 2025-10-09 13:50:30,799 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:50:31,207 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:50:38,912 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-78394098:ctest-vn-78394098 2025-10-09 13:50:44,684 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:7f08033d-5437-4394-acfd-277dc4c22383 2025-10-09 13:50:44,739 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-78394098:ctest-vn-78394098 2025-10-09 13:50:44,795 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:7f08033d-5437-4394-acfd-277dc4c22383 2025-10-09 13:50:56,616 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5b30b4d0-fc19-4f4e-a44f-7f5592f2f2db 2025-10-09 13:50:56,671 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-78394098:ctest-vn-78394098 2025-10-09 13:50:56,719 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5b30b4d0-fc19-4f4e-a44f-7f5592f2f2db 2025-10-09 13:50:58,980 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f9bdb7fc-78b2-4019-882d-fd7bd3bbaae4 2025-10-09 13:50:59,036 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-78394098:ctest-vn-78394098 2025-10-09 13:50:59,087 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:f9bdb7fc-78b2-4019-882d-fd7bd3bbaae4 2025-10-09 13:51:18,328 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VnListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-78394098 2025-10-09 13:51:18,379 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name=admin_domain:ctest-TestBasicVMVN-28715814:ctest-vn-78394098:ctest-vn-78394098 2025-10-09 13:51:18,431 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_VrfListReq?name= 2025-10-09 13:51:18,479 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_KRouteReq?x=2 2025-10-09 13:51:18,531 - DEBUG - Unable to fetch route details in vrouter for vrf index 2, Got :[] 2025-10-09 13:51:19,184 - DEBUG - Requesting: https://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-09 13:53:17,023 - INFO - Using existing project ['admin_domain', 'admin'](2f0ea955-6c00-46c1-8ddd-92c9167f91d0) 2025-10-09 13:53:17,032 - INFO - ================================================================================ 2025-10-09 13:53:17,032 - INFO - STARTING TEST : test_contrail_status 2025-10-09 13:53:17,032 - INFO - TEST DESCRIPTION : Test to verify that all services are running and active 2025-10-09 13:53:17,268 - DEBUG - Skipping xmpp flap check 2025-10-09 13:53:17,269 - INFO - Initial checks done. Running the testcase now 2025-10-09 13:53:17,269 - INFO - 2025-10-09 13:53:17,387 - INFO - 10.0.0.41 2025-10-09 13:53:17,387 - INFO - config 2025-10-09 13:53:17,444 - INFO - config-nodemgr:active 2025-10-09 13:53:17,499 - INFO - api-server:active 2025-10-09 13:53:17,555 - INFO - schema:active 2025-10-09 13:53:17,611 - INFO - svc-monitor:active 2025-10-09 13:53:17,667 - INFO - device-manager:active 2025-10-09 13:53:17,667 - INFO - config-database 2025-10-09 13:53:17,667 - INFO - config-cassandra:active 2025-10-09 13:53:17,667 - INFO - config-zookeeper:active 2025-10-09 13:53:17,668 - INFO - config-rabbitmq:active 2025-10-09 13:53:17,668 - INFO - vrouter 2025-10-09 13:53:17,723 - INFO - vrouter-nodemgr:active 2025-10-09 13:53:17,779 - INFO - agent:active 2025-10-09 13:53:17,779 - INFO - control 2025-10-09 13:53:17,835 - INFO - control-nodemgr:active 2025-10-09 13:53:17,891 - INFO - control:active 2025-10-09 13:53:17,891 - INFO - named:active 2025-10-09 13:53:17,947 - INFO - dns:active 2025-10-09 13:53:17,947 - INFO - analytics 2025-10-09 13:53:18,003 - INFO - analytics-nodemgr:active 2025-10-09 13:53:18,059 - INFO - analytics-api:active 2025-10-09 13:53:18,115 - INFO - collector:active 2025-10-09 13:53:18,115 - INFO - analytics-database 2025-10-09 13:53:18,115 - INFO - analytics-cassandra:active 2025-10-09 13:53:18,171 - INFO - analyticsdb-nodemgr:active 2025-10-09 13:53:18,223 - INFO - query-engine:active 2025-10-09 13:53:18,223 - INFO - webui 2025-10-09 13:53:18,223 - INFO - webui:active 2025-10-09 13:53:18,223 - INFO - webui-middleware:active 2025-10-09 13:53:18,223 - INFO - redis:active 2025-10-09 13:53:18,223 - INFO - analytics_snmp 2025-10-09 13:53:18,278 - INFO - snmp-collector:active 2025-10-09 13:53:18,279 - INFO - snmp-topology:active 2025-10-09 13:53:18,279 - INFO - analytics_alarm 2025-10-09 13:53:18,279 - INFO - alarmgen:active 2025-10-09 13:53:18,279 - INFO - All the contrail services are up on all nodes 2025-10-09 13:53:18,279 - INFO - contrail-status passed 2025-10-09 13:53:18,502 - DEBUG - Skipping xmpp flap check
2025-10-09 13:53:18,502 - INFO - END TEST : test_contrail_status : PASSED[0:00:01]
2025-10-09 13:53:18,502 - INFO - --------------------------------------------------------------------------------