2026-04-22 11:26:54,112 - DEBUG - [10.0.0.249]: Running cmd : hostname 2026-04-22 11:26:54,437 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-397-1 2026-04-22 11:26:54,438 - DEBUG - [10.0.0.249]: Running cmd : hostname -f 2026-04-22 11:26:54,577 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-397-1.cluster.local 2026-04-22 11:26:54,577 - DEBUG - [10.0.0.249]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-04-22 11:26:54,780 - DEBUG - Output : NAMES contrail_test_CgPWYuIah k8s_autoscaler_dns-autoscaler-595558c478-nvqb8_kube-system_6d9244f2-230b-431a-aaa5-3c3dc31e94c4_0 k8s_coredns_coredns-77f7cc69db-ns2vx_kube-system_8b970670-d09d-435f-a467-9d8d104abbbe_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-pc4pz_kube-system_57bc8fdd-ff55-4027-8307-811c1ec77b0b_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-pc4pz_kube-system_57bc8fdd-ff55-4027-8307-811c1ec77b0b_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-pc4pz_kube-system_57bc8fdd-ff55-4027-8307-811c1ec77b0b_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-cxgzf_kube-system_bccb10b5-29e1-4b07-99eb-b51ec1690e2e_0 k8s_opensdn-controller-control-named_opensdn-controller-control-cxgzf_kube-system_bccb10b5-29e1-4b07-99eb-b51ec1690e2e_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-l5ths_kube-system_9014c7ad-76ad-4a60-bbfe-5905b34c16ac_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-w99gh_kube-system_5d7d5b67-ccbd-4bf4-9789-8864e91577cd_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-w99gh_kube-system_5d7d5b67-ccbd-4bf4-9789-8864e91577cd_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-56582_kube-system_c77f2c1a-929a-45e6-b3d6-1ab096a4b51d_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-56582_kube-system_c77f2c1a-929a-45e6-b3d6-1ab096a4b51d_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-pc4pz_kube-system_57bc8fdd-ff55-4027-8307-811c1ec77b0b_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-mzk4m_kube-system_6f7b47b1-3ac7-45e6-a8e8-4d67187b312a_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-mkt65_kube-system_0029fbca-1d2a-4709-a1d2-dd428c62bc2f_0 k8s_opensdn-analytics-collector_opensdn-analytics-mkt65_kube-system_0029fbca-1d2a-4709-a1d2-dd428c62bc2f_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-cxgzf_kube-system_bccb10b5-29e1-4b07-99eb-b51ec1690e2e_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-l5ths_kube-system_9014c7ad-76ad-4a60-bbfe-5905b34c16ac_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-w99gh_kube-system_5d7d5b67-ccbd-4bf4-9789-8864e91577cd_0 k8s_kafka_opensdn-analytics-alarm-56582_kube-system_c77f2c1a-929a-45e6-b3d6-1ab096a4b51d_0 k8s_opensdn-controller-config-api_opensdn-controller-config-pc4pz_kube-system_57bc8fdd-ff55-4027-8307-811c1ec77b0b_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-mzk4m_kube-system_6f7b47b1-3ac7-45e6-a8e8-4d67187b312a_0 k8s_opensdn-analytics-api_opensdn-analytics-mkt65_kube-system_0029fbca-1d2a-4709-a1d2-dd428c62bc2f_0 k8s_opensdn-controller-control_opensdn-controller-control-cxgzf_kube-system_bccb10b5-29e1-4b07-99eb-b51ec1690e2e_0 k8s_config-zookeeper_config-zookeeper-5zdnr_kube-system_177e5890-2fc2-4972-b2b9-84a362294502_0 k8s_rabbitmq_rabbitmq-6hrx5_kube-system_9c8ce2e0-59cc-497e-875c-d07f126ec18c_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-8lpjd_kube-system_5aea2d71-5429-4225-83d5-233960e90bc3_0 k8s_opensdn-configdb_opensdn-configdb-8lpjd_kube-system_5aea2d71-5429-4225-83d5-233960e90bc3_0 k8s_redis_redis-ft2rz_kube-system_f8321a9e-c218-48aa-b2d4-e7b4589949de_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-l5ths_kube-system_9014c7ad-76ad-4a60-bbfe-5905b34c16ac_0 k8s_opensdn-agent-nodemgr_opensdn-agent-h9td6_kube-system_33c5eb72-f7fc-4c9e-9466-7e26249bc62b_0 k8s_opensdn-vrouter-agent_opensdn-agent-h9td6_kube-system_33c5eb72-f7fc-4c9e-9466-7e26249bc62b_0 k8s_opensdn-kube-manager_opensdn-kube-manager-nj6rv_kube-system_5773573f-e011-43c6-89cb-4e4b767b2191_0 k8s_kube-proxy_kube-proxy-czz97_kube-system_b07b4c60-4bdf-4959-8e5b-bbaee49e0191_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-397-1_kube-system_052048e2c12e49f9db9181848dc8cabe_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-397-1_kube-system_e2962e5f254c521e92fd8d1845b12763_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-397-1_kube-system_b89c5cf6fb38000782b06abf8f027e02_0 2026-04-22 11:26:54,781 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-04-22 11:26:54,914 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.249/24 2026-04-22 11:26:54,915 - DEBUG - [10.0.0.249]: Running cmd : getent hosts 10.0.0.249 | head -n 1 | awk '{print $2}' 2026-04-22 11:26:55,053 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-397-1.cluster.local 2026-04-22 11:26:55,053 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-04-22 11:26:55,199 - DEBUG - Output : 10.0.0.249/24 2026-04-22 11:26:55,601 - INFO - Using existing project ['default-domain', 'k8s-default'](68a975af-b6e3-41a4-884f-e988d80139a1) 2026-04-22 11:26:55,726 - INFO - Using existing project ['default-domain', 'k8s-default'](68a975af-b6e3-41a4-884f-e988d80139a1) 2026-04-22 11:26:55,873 - INFO - ================================================================================ 2026-04-22 11:26:55,873 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-04-22 11:26:55,874 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-04-22 11:26:56,209 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2026-04-22 11:26:56,218 - DEBUG - Nothing to compare xmpp stats {'10.0.0.249': {'10.0.0.249': '2'}} with 2026-04-22 11:26:56,218 - INFO - Initial checks done. Running the testcase now 2026-04-22 11:26:56,218 - INFO - 2026-04-22 11:26:56,229 - INFO - Namespace default already exists 2026-04-22 11:26:56,229 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-nj6rv_kube-system_5773573f-e011-43c6-89cb-4e4b767b2191_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-04-22 11:26:56,428 - DEBUG - Output : cluster_project={} 2026-04-22 11:26:56,515 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-04-22 11:26:56,515 - DEBUG - Project uuid in Contrail is 68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:26:56,515 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-nj6rv_kube-system_5773573f-e011-43c6-89cb-4e4b767b2191_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-04-22 11:26:56,712 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-04-22 11:26:56,713 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_NamespaceDatabaseList?namespace_uuid=cc7707d4-6191-44d2-999d-71cdc784e94b 2026-04-22 11:26:56,718 - INFO - Namespace default with uuid cc7707d4-6191-44d2-999d-71cdc784e94b found in kube manager 2026-04-22 11:26:56,718 - INFO - Namespace default verification passed 2026-04-22 11:26:56,721 - DEBUG - Service ctest-nginx-svc-91528853 not present 2026-04-22 11:26:56,722 - INFO - Creating service ctest-nginx-svc-91528853 2026-04-22 11:26:56,750 - INFO - Created Service ctest-nginx-svc-91528853 2026-04-22 11:26:56,766 - DEBUG - Pod ctest-nginx-pod-30697145 not present 2026-04-22 11:26:56,767 - INFO - Creating Pod ctest-nginx-pod-30697145 2026-04-22 11:26:56,806 - DEBUG - Pod : ctest-nginx-pod-30697145 UUID is 7b9622ca-637f-4cf8-bb2e-53b639b4d822 2026-04-22 11:26:56,811 - DEBUG - Pod ctest-nginx-pod-08448639 not present 2026-04-22 11:26:56,811 - INFO - Creating Pod ctest-nginx-pod-08448639 2026-04-22 11:26:56,849 - DEBUG - Pod : ctest-nginx-pod-08448639 UUID is 053dc586-96aa-48f6-8681-c06c4cf45bcb 2026-04-22 11:26:57,289 - INFO - Created VN __public__, UUID :de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:26:57,666 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-22 11:26:57,686 - DEBUG - Requesting: http://10.0.0.249:8082/domain/b4fc03f4-68b3-4f44-ab12-6593943ead22 2026-04-22 11:26:57,708 - DEBUG - Requesting: http://10.0.0.249:8082/project/68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:26:57,834 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:26:57,857 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:26:57,895 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:26:57,917 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:26:57,928 - DEBUG - Route Targets: [] 2026-04-22 11:26:57,928 - DEBUG - RT names not yet present for VN __public__ 2026-04-22 11:27:02,929 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-22 11:27:02,936 - DEBUG - Requesting: http://10.0.0.249:8082/domain/b4fc03f4-68b3-4f44-ab12-6593943ead22 2026-04-22 11:27:02,947 - DEBUG - Requesting: http://10.0.0.249:8082/project/68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:27:03,017 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,030 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,045 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:27:03,056 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:27:03,065 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/d1d9bbf8-5bcb-4f20-9b89-aa4c919bb263 2026-04-22 11:27:03,073 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-04-22 11:27:03,073 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,085 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:27:03,104 - INFO - Verified VN network id 11 for VN de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,104 - INFO - Verifications in API Server for VN __public__ passed 2026-04-22 11:27:03,104 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,115 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:27:03,123 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:27:03,131 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/d1d9bbf8-5bcb-4f20-9b89-aa4c919bb263 2026-04-22 11:27:03,141 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-04-22 11:27:03,152 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16014401503311643772', 'uuid-lslong': '10834937130520592334'}, 'enable': 'true', 'created': '2026-04-22T11:26:57', 'last-modified': '2026-04-22T11:26:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.602590'} 2026-04-22 11:27:03,152 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:27:03,168 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-04-22 11:27:03,169 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-04-22 11:27:03,170 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-04-22 11:27:03,170 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-22 11:27:03,178 - DEBUG - Requesting: http://10.0.0.249:8082/domain/b4fc03f4-68b3-4f44-ab12-6593943ead22 2026-04-22 11:27:03,190 - DEBUG - Requesting: http://10.0.0.249:8082/project/68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:27:03,252 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,264 - DEBUG - =>VN __public__ has no policy to be verified 2026-04-22 11:27:03,264 - DEBUG - Verifying the vn in opserver 2026-04-22 11:27:03,264 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.249 2026-04-22 11:27:03,264 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-04-22 11:27:03,271 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-04-22 11:27:03,271 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-04-22 11:27:03,272 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:27:03,280 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:27:03,286 - DEBUG - Do not have enough data to verify VN in agent 2026-04-22 11:27:03,286 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:27:03,293 - DEBUG - VRF ids for VN __public__: {} 2026-04-22 11:27:03,478 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-04-22 11:27:03,705 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-22 11:27:03,712 - DEBUG - Requesting: http://10.0.0.249:8082/domain/b4fc03f4-68b3-4f44-ab12-6593943ead22 2026-04-22 11:27:03,720 - DEBUG - Requesting: http://10.0.0.249:8082/project/68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:27:03,753 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,775 - DEBUG - Requesting: http://10.0.0.249:8082/floating-ip-pool/f156d499-5917-4c39-a521-ddd8e8d3bb1c 2026-04-22 11:27:03,790 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-22 11:27:03,799 - DEBUG - Requesting: http://10.0.0.249:8082/domain/b4fc03f4-68b3-4f44-ab12-6593943ead22 2026-04-22 11:27:03,810 - DEBUG - Requesting: http://10.0.0.249:8082/project/68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:27:03,928 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/de3e9551-bf5f-447c-965d-6dea0c5ebfce 2026-04-22 11:27:03,945 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-04-22 11:27:03,945 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-04-22 11:27:03,961 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-04-22 11:27:03,974 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-04-22 11:27:03,974 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-04-22 11:27:03,982 - DEBUG - Ingress ctest-nginx-ingress-69214565 not present 2026-04-22 11:27:03,982 - INFO - Creating Ingress ctest-nginx-ingress-69214565 2026-04-22 11:27:03,993 - INFO - Created Ingress ctest-nginx-ingress-69214565 2026-04-22 11:27:04,004 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-69214565 2026-04-22 11:27:07,011 - DEBUG - For Ingress ctest-nginx-ingress-69214565, Cluster IP: 10.47.255.248, External IPs ['116.145.48.252'] 2026-04-22 11:27:07,011 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-69214565 2026-04-22 11:27:07,011 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=66f2eaca-7443-453a-8799-ff962c5b83e6 2026-04-22 11:27:07,016 - INFO - Ingress ctest-nginx-ingress-69214565 with uuid 66f2eaca-7443-453a-8799-ff962c5b83e6 found in kube manager 2026-04-22 11:27:07,017 - INFO - Ingress ctest-nginx-ingress-69214565 verification passed 2026-04-22 11:27:07,022 - DEBUG - Pod ctest-busybox-pod-01318217 not present 2026-04-22 11:27:07,023 - INFO - Creating Pod ctest-busybox-pod-01318217 2026-04-22 11:27:07,065 - DEBUG - Pod : ctest-busybox-pod-01318217 UUID is 38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:07,076 - DEBUG - Pod ctest-nginx-pod-30697145 not in running state.Currently in Pending 2026-04-22 11:27:12,086 - DEBUG - Pod ctest-nginx-pod-30697145 not in running state.Currently in Pending 2026-04-22 11:27:17,102 - DEBUG - Pod ctest-nginx-pod-30697145 not in running state.Currently in Pending 2026-04-22 11:27:22,111 - INFO - Pod ctest-nginx-pod-30697145 is in running state.Got IP 10.47.255.250 2026-04-22 11:27:22,171 - DEBUG - Pod ctest-nginx-pod-30697145 has vmi 2e1403c8-3e3e-11f1-8cf8-0242448ae2c7 2026-04-22 11:27:22,171 - INFO - Verified pod ctest-nginx-pod-30697145 in contrail-api 2026-04-22 11:27:22,171 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-04-22 11:27:22,183 - DEBUG - VMI 2e1403c8-3e3e-11f1-8cf8-0242448ae2c7 is active in agent 10.0.0.249 2026-04-22 11:27:22,183 - INFO - Verified Pod ctest-nginx-pod-30697145 in agent 10.0.0.249 2026-04-22 11:27:22,183 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=7b9622ca-637f-4cf8-bb2e-53b639b4d822 2026-04-22 11:27:22,190 - INFO - Pod ctest-nginx-pod-30697145 with uuid 7b9622ca-637f-4cf8-bb2e-53b639b4d822 found in kube manager 2026-04-22 11:27:22,191 - INFO - Pod ctest-nginx-pod-30697145 verification passed 2026-04-22 11:27:22,297 - DEBUG - [Pod ctest-nginx-pod-30697145] Cmd: echo ctest-nginx-pod-30697145 > /usr/share/nginx/html/index.html, Output: 2026-04-22 11:27:22,309 - INFO - Pod ctest-nginx-pod-08448639 is in running state.Got IP 10.47.255.249 2026-04-22 11:27:22,365 - DEBUG - Pod ctest-nginx-pod-08448639 has vmi 2eddcc08-3e3e-11f1-8cf8-0242448ae2c7 2026-04-22 11:27:22,365 - INFO - Verified pod ctest-nginx-pod-08448639 in contrail-api 2026-04-22 11:27:22,366 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-04-22 11:27:22,376 - DEBUG - VMI 2eddcc08-3e3e-11f1-8cf8-0242448ae2c7 is active in agent 10.0.0.249 2026-04-22 11:27:22,376 - INFO - Verified Pod ctest-nginx-pod-08448639 in agent 10.0.0.249 2026-04-22 11:27:22,376 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=053dc586-96aa-48f6-8681-c06c4cf45bcb 2026-04-22 11:27:22,382 - INFO - Pod ctest-nginx-pod-08448639 with uuid 053dc586-96aa-48f6-8681-c06c4cf45bcb found in kube manager 2026-04-22 11:27:22,382 - INFO - Pod ctest-nginx-pod-08448639 verification passed 2026-04-22 11:27:22,498 - DEBUG - [Pod ctest-nginx-pod-08448639] Cmd: echo ctest-nginx-pod-08448639 > /usr/share/nginx/html/index.html, Output: 2026-04-22 11:27:22,507 - DEBUG - Pod ctest-busybox-pod-01318217 not in running state.Currently in Pending 2026-04-22 11:27:27,517 - INFO - Pod ctest-busybox-pod-01318217 is in running state.Got IP 10.47.255.247 2026-04-22 11:27:27,574 - DEBUG - Pod ctest-busybox-pod-01318217 has vmi 33a62b68-3e3e-11f1-8cf8-0242448ae2c7 2026-04-22 11:27:27,575 - INFO - Verified pod ctest-busybox-pod-01318217 in contrail-api 2026-04-22 11:27:27,575 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-04-22 11:27:27,590 - DEBUG - VMI 33a62b68-3e3e-11f1-8cf8-0242448ae2c7 is active in agent 10.0.0.249 2026-04-22 11:27:27,590 - INFO - Verified Pod ctest-busybox-pod-01318217 in agent 10.0.0.249 2026-04-22 11:27:27,590 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:27,596 - INFO - Pod ctest-busybox-pod-01318217 with uuid 38b368ca-2301-442e-894c-55107c6d4ec9 found in kube manager 2026-04-22 11:27:27,596 - INFO - Pod ctest-busybox-pod-01318217 verification passed 2026-04-22 11:27:27,704 - DEBUG - [Pod ctest-busybox-pod-01318217] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-08448639 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-04-22 11:27:27,704 - DEBUG - [Pod Pod ctest-busybox-pod-01318217] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-04-22 11:27:27,704 - DEBUG - [Pod Pod ctest-busybox-pod-01318217] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout ctest-nginx-pod-08448639 - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-04-22 11:27:27,809 - DEBUG - [Pod ctest-busybox-pod-01318217] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-30697145 2026-04-22 11:27:27,809 - DEBUG - [Pod Pod ctest-busybox-pod-01318217] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-04-22 11:27:27,809 - DEBUG - [Pod Pod ctest-busybox-pod-01318217] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-30697145 2026-04-22 11:27:27,809 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-30697145': 1, 'ctest-nginx-pod-08448639': 1} 2026-04-22 11:27:27,810 - WARNING - Nothing to delete parallely 2026-04-22 11:27:27,810 - INFO - Deleting pod default:ctest-busybox-pod-01318217 2026-04-22 11:27:27,822 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:27,834 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:27:32,835 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:32,841 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:27:37,842 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:37,860 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:27:42,861 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:42,869 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:27:47,870 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:47,876 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:27:52,877 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:52,884 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:27:57,885 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:27:57,893 - WARNING - Pod uuid 38b368ca-2301-442e-894c-55107c6d4ec9 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:28:02,894 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=38b368ca-2301-442e-894c-55107c6d4ec9 2026-04-22 11:28:02,902 - DEBUG - Pod 38b368ca-2301-442e-894c-55107c6d4ec9 is not in agent 10.0.0.249 VM list 2026-04-22 11:28:02,902 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:33a62b68-3e3e-11f1-8cf8-0242448ae2c7 2026-04-22 11:28:02,909 - DEBUG - VMI 33a62b68-3e3e-11f1-8cf8-0242448ae2c7 is removed from agent 10.0.0.249 2026-04-22 11:28:02,909 - INFO - Verified that pod ctest-busybox-pod-01318217 is removed in agent 2026-04-22 11:28:02,911 - INFO - Deleting Ingress : ctest-nginx-ingress-69214565 2026-04-22 11:28:02,919 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=66f2eaca-7443-453a-8799-ff962c5b83e6 2026-04-22 11:28:02,928 - ERROR - Ingress ctest-nginx-ingress-69214565 with uuid 66f2eaca-7443-453a-8799-ff962c5b83e6 still found in kube manager 2026-04-22 11:28:03,928 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=66f2eaca-7443-453a-8799-ff962c5b83e6 2026-04-22 11:28:03,940 - ERROR - Ingress ctest-nginx-ingress-69214565 with uuid 66f2eaca-7443-453a-8799-ff962c5b83e6 still found in kube manager 2026-04-22 11:28:04,941 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=66f2eaca-7443-453a-8799-ff962c5b83e6 2026-04-22 11:28:04,947 - INFO - Ingress ctest-nginx-ingress-69214565 with uuid 66f2eaca-7443-453a-8799-ff962c5b83e6 deleted successfully from kube manager 2026-04-22 11:28:04,947 - INFO - Verifications on Ingress %s cleanup passed 2026-04-22 11:28:04,948 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-04-22 11:28:05,167 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-04-22 11:28:05,191 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-04-22 11:28:05,191 - INFO - Deleting VN __public__ 2026-04-22 11:28:05,438 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/bb2b0290-11ca-4267-b044-56b26c0e9b34 2026-04-22 11:28:05,451 - DEBUG - Response Code: 404 2026-04-22 11:28:05,451 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-04-22 11:28:05,464 - DEBUG - Requesting: http://10.0.0.249:8082/domain/b4fc03f4-68b3-4f44-ab12-6593943ead22 2026-04-22 11:28:05,482 - DEBUG - Requesting: http://10.0.0.249:8082/project/68a975af-b6e3-41a4-884f-e988d80139a1 2026-04-22 11:28:05,539 - INFO - Validated that VN __public__ is not found in API Server 2026-04-22 11:28:05,539 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-04-22 11:28:05,548 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:28:05,563 - DEBUG - VN __public__ is not present in Agent 10.0.0.249 2026-04-22 11:28:05,563 - INFO - Validated that VN __public__ is not in any agent 2026-04-22 11:28:05,564 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:28:05,576 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-22 11:28:05,590 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-04-22 11:28:05,610 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-04-22 11:28:05,611 - INFO - Deleting pod default:ctest-nginx-pod-08448639 2026-04-22 11:28:05,624 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=053dc586-96aa-48f6-8681-c06c4cf45bcb 2026-04-22 11:28:05,637 - WARNING - Pod uuid 053dc586-96aa-48f6-8681-c06c4cf45bcb is still seen in agent 10.0.0.249 VM list 2026-04-22 11:28:10,638 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=053dc586-96aa-48f6-8681-c06c4cf45bcb 2026-04-22 11:28:10,645 - DEBUG - Pod 053dc586-96aa-48f6-8681-c06c4cf45bcb is not in agent 10.0.0.249 VM list 2026-04-22 11:28:10,645 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2eddcc08-3e3e-11f1-8cf8-0242448ae2c7 2026-04-22 11:28:10,651 - DEBUG - VMI 2eddcc08-3e3e-11f1-8cf8-0242448ae2c7 is removed from agent 10.0.0.249 2026-04-22 11:28:10,651 - INFO - Verified that pod ctest-nginx-pod-08448639 is removed in agent 2026-04-22 11:28:10,652 - INFO - Deleting pod default:ctest-nginx-pod-30697145 2026-04-22 11:28:10,672 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=7b9622ca-637f-4cf8-bb2e-53b639b4d822 2026-04-22 11:28:10,679 - WARNING - Pod uuid 7b9622ca-637f-4cf8-bb2e-53b639b4d822 is still seen in agent 10.0.0.249 VM list 2026-04-22 11:28:15,680 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=7b9622ca-637f-4cf8-bb2e-53b639b4d822 2026-04-22 11:28:15,687 - DEBUG - Pod 7b9622ca-637f-4cf8-bb2e-53b639b4d822 is not in agent 10.0.0.249 VM list 2026-04-22 11:28:15,687 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2e1403c8-3e3e-11f1-8cf8-0242448ae2c7 2026-04-22 11:28:15,692 - DEBUG - VMI 2e1403c8-3e3e-11f1-8cf8-0242448ae2c7 is removed from agent 10.0.0.249 2026-04-22 11:28:15,692 - INFO - Verified that pod ctest-nginx-pod-30697145 is removed in agent 2026-04-22 11:28:15,693 - INFO - Deleting service : ctest-nginx-svc-91528853 2026-04-22 11:28:16,336 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2026-04-22 11:28:16,342 - DEBUG - No XMPP flaps were noticed during the test