2025-03-27 20:06:09,272 - DEBUG - [10.0.0.17]: Running cmd : hostname 2025-03-27 20:06:09,547 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-293-1.novalocal 2025-03-27 20:06:09,547 - DEBUG - [10.0.0.17]: Running cmd : hostname -f 2025-03-27 20:06:09,662 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-293-1.novalocal 2025-03-27 20:06:09,662 - DEBUG - [10.0.0.17]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-03-27 20:06:09,838 - DEBUG - Output : NAMES contrail_test_5d9aZ2sbv vrouter_vrouter-agent_1 vrouter_nodemgr_1 vrouter_node-init_1 rsyslogd_rsyslogd_1 kubemanager_k8s_kubemanager_1 kubemanager_k8s_node-init_1 cni_vrouter-cni_1 analytics_snmp_nodemgr_1 analytics_snmp_topology_1 analytics_snmp_snmp-collector_1 analytics_snmp_node-init_1 analytics_alarm_alarm-gen_1 analytics_alarm_nodemgr_1 analytics_alarm_node-init_1 analytics_alarm_kafka_1 analytics_collector_1 analytics_api_1 analytics_nodemgr_1 analytics_node-init_1 analytics_database_query-engine_1 analytics_database_nodemgr_1 analytics_database_cassandra_1 analytics_database_node-init_1 control_named_1 control_nodemgr_1 control_control_1 control_dns_1 control_node-init_1 webui_job_1 webui_web_1 webui_node-init_1 config_devicemgr_1 config_svcmonitor_1 config_nodemgr_1 config_api_1 config_schema_1 config_dnsmasq_1 config_node-init_1 config_database_rabbitmq_1 config_database_nodemgr_1 config_database_zookeeper_1 config_database_node-init_1 config_database_cassandra_1 redis_redis_1 2025-03-27 20:06:09,838 - DEBUG - [10.0.0.17]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-03-27 20:06:09,954 - DEBUG - Output : 172.17.0.1/16 10.0.0.17/24 2025-03-27 20:06:09,954 - DEBUG - [10.0.0.17]: Running cmd : getent hosts 10.0.0.17 | head -n 1 | awk '{print $2}' 2025-03-27 20:06:10,091 - DEBUG - Output : cn-jenkins-deploy-platform-ansible-k8s-293-1.novalocal 2025-03-27 20:06:10,091 - DEBUG - [10.0.0.17]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-03-27 20:06:10,214 - DEBUG - Output : 10.0.0.17/24 2025-03-27 20:06:10,496 - INFO - Using existing project ['default-domain', 'k8s-default'](12d4089e-12d2-4e6c-b369-b5a89708c749) 2025-03-27 20:06:10,606 - INFO - Using existing project ['default-domain', 'k8s-default'](12d4089e-12d2-4e6c-b369-b5a89708c749) 2025-03-27 20:06:10,759 - INFO - ================================================================================ 2025-03-27 20:06:10,760 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-03-27 20:06:10,760 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-03-27 20:06:10,993 - DEBUG - Requesting: http://10.0.0.17:8085/Snh_AgentXmppConnectionStatusReq? 2025-03-27 20:06:11,000 - DEBUG - Nothing to compare xmpp stats {'10.0.0.17': {'10.0.0.17': '0'}} with 2025-03-27 20:06:11,000 - INFO - Initial checks done. Running the testcase now 2025-03-27 20:06:11,000 - INFO - 2025-03-27 20:06:11,005 - INFO - Namespace default already exists 2025-03-27 20:06:11,005 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-03-27 20:06:11,145 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:11,192 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-03-27 20:06:11,192 - DEBUG - Project uuid in Contrail is 12d4089e-12d2-4e6c-b369-b5a89708c749 2025-03-27 20:06:11,192 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:11,299 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:14,300 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:14,407 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:17,408 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:17,515 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:20,516 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:20,648 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:23,650 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:23,767 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:26,768 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:26,897 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:29,899 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:30,006 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:33,007 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:33,139 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:36,139 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:36,257 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:39,258 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:39,366 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:42,368 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:42,490 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:42,498 - WARNING - Nothing to delete parallely 2025-03-27 20:06:42,758 - ERROR - AssertionError Python 3.6.8: /usr/bin/python3 Thu Mar 27 20:06:42 2025 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /contrail-test/tcutils/wrappers.py in wrapper(self=, *args=(), **kwargs={}) 78 log.info('Initial checks done. Running the testcase now') 79 log.info('') 80 result = function(self, *args, **kwargs) 81 if self.inputs.upgrade: 82 pid = os.getpid() result = None function = self = args = () kwargs = {} /contrail-test/scripts/k8s_scripts/test_ingress.py in test_ingress_ip_assignment(self=) 39 labels = {'app':app} 40 namespace = self.setup_namespace(name='default') 41 assert namespace.verify_on_setup() 42 43 service = self.setup_http_service(namespace=namespace.name, namespace = namespace.verify_on_setup = > /contrail-test/fixtures/k8s/namespace.py in verify_on_setup(self=) 51 self.name)) 52 return False 53 if not self.verify_namespace_in_kube_manager(): 54 self.logger.error('Namespace %s not seen in Kube Manager' % ( 55 self.name)) self = self.verify_namespace_in_kube_manager = .deco_retry..f_retry of > /contrail-test/tcutils/util.py in f_retry(*args=(,), **kwargs={}) 82 mtries, mdelay = tries, delay # make mutable 83 84 result = f(*args, **kwargs) # first attempt 85 rv = result 86 final = False result undefined f = args = (,) kwargs = {} /contrail-test/fixtures/k8s/namespace.py in verify_namespace_in_kube_manager(self=) 119 @retry(delay=2, tries=10) 120 def verify_namespace_in_kube_manager(self): 121 km_h = self.connections.get_kube_manager_h() 122 self.namespace_info = km_h.get_namespace_info(ns_uuid = self.uuid) 123 if self.namespace_info: km_h undefined self = self.connections = self.connections.get_kube_manager_h = > /contrail-test/common/connections.py in get_kube_manager_h(self=, refresh=False) 306 self._get_kube_manager_h(refresh=refresh) 307 msg = "Kubernetes manager service is not up" 308 assert self._kube_manager_inspect is not None, msg 309 return self._kube_manager_inspect 310 self = self._kube_manager_inspect = None msg = 'Kubernetes manager service is not up' AssertionError: Kubernetes manager service is not up __cause__ = None __class__ = __context__ = None __delattr__ = __dict__ = {} __dir__ = __doc__ = 'Assertion failed.' __eq__ = __format__ = __ge__ = __getattribute__ = __gt__ = __hash__ = __init__ = __init_subclass__ = __le__ = __lt__ = __ne__ = __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __suppress_context__ = False __traceback__ = args = ('Kubernetes manager service is not up',) with_traceback = The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/contrail-test/tcutils/wrappers.py", line 80, in wrapper result = function(self, *args, **kwargs) File "/contrail-test/scripts/k8s_scripts/test_ingress.py", line 41, in test_ingress_ip_assignment assert namespace.verify_on_setup() File "/contrail-test/fixtures/k8s/namespace.py", line 53, in verify_on_setup if not self.verify_namespace_in_kube_manager(): File "/contrail-test/tcutils/util.py", line 84, in f_retry result = f(*args, **kwargs) # first attempt File "/contrail-test/fixtures/k8s/namespace.py", line 121, in verify_namespace_in_kube_manager km_h = self.connections.get_kube_manager_h() File "/contrail-test/common/connections.py", line 308, in get_kube_manager_h assert self._kube_manager_inspect is not None, msg AssertionError: Kubernetes manager service is not up 2025-03-27 20:06:42,759 - DEBUG - Requesting: http://10.0.0.17:8085/Snh_AgentXmppConnectionStatusReq? 2025-03-27 20:06:42,766 - DEBUG - No XMPP flaps were noticed during the test 2025-03-27 20:06:42,766 - INFO -
2025-03-27 20:06:42,767 - INFO - END TEST : test_ingress_ip_assignment : FAILED[0:00:32]
2025-03-27 20:06:42,767 - INFO - -------------------------------------------------------------------------------- 2025-03-27 20:06:49,142 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-03-27 20:06:49,253 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:49,292 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:49,400 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:52,402 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:52,543 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:55,544 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:55,726 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:06:58,727 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:06:58,847 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:01,847 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:01,968 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:04,969 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:05,086 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:08,086 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:08,193 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:11,194 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:11,367 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:14,368 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:14,508 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:17,508 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:17,629 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running 2025-03-27 20:07:20,630 - DEBUG - [10.0.0.17]: Running cmd : docker exec --privileged -it kubemanager_k8s_node-init_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-03-27 20:07:20,764 - DEBUG - Output : Error response from daemon: container f76f78ae6f4f1ca8949fc412f04f6ae4e4cfda73ecccc9608fbc6262a178e746 is not running