2026-02-20 12:18:39,389 - DEBUG - [10.0.0.240]: Running cmd : hostname 2026-02-20 12:18:39,632 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-331-1 2026-02-20 12:18:39,632 - DEBUG - [10.0.0.240]: Running cmd : hostname -f 2026-02-20 12:18:39,757 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-331-1.cluster.local 2026-02-20 12:18:39,757 - DEBUG - [10.0.0.240]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-20 12:18:39,926 - DEBUG - Output : NAMES contrail_test_7SokZBsZE k8s_autoscaler_dns-autoscaler-595558c478-wpvbn_kube-system_f6bed2b5-302f-4d0f-9346-4aa1bd8d3e06_0 k8s_coredns_coredns-77f7cc69db-vktc7_kube-system_7e55edce-5358-4cbd-abc7-1c31b70d015b_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-fc4qg_kube-system_476c310f-5870-4691-b381-1ac8fa51ec8c_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-fc4qg_kube-system_476c310f-5870-4691-b381-1ac8fa51ec8c_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-fc4qg_kube-system_476c310f-5870-4691-b381-1ac8fa51ec8c_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-5f4z5_kube-system_5bd1373d-fdab-49de-85cf-8b5569f2cfd3_0 k8s_opensdn-controller-control-named_opensdn-controller-control-5f4z5_kube-system_5bd1373d-fdab-49de-85cf-8b5569f2cfd3_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-6v6c4_kube-system_07b4d3f6-5895-4537-9573-fb44387b4542_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-g48fv_kube-system_dd9d4c9c-7556-4228-b225-302c5ec45374_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-g48fv_kube-system_dd9d4c9c-7556-4228-b225-302c5ec45374_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-fc4qg_kube-system_476c310f-5870-4691-b381-1ac8fa51ec8c_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-rdt7g_kube-system_b3cc335e-b7c8-40e5-ab3a-eac5dfa9fdbb_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-5f4z5_kube-system_5bd1373d-fdab-49de-85cf-8b5569f2cfd3_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-jczm4_kube-system_dce1bcc7-4746-4679-8173-306428087321_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-jczm4_kube-system_dce1bcc7-4746-4679-8173-306428087321_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-6v6c4_kube-system_07b4d3f6-5895-4537-9573-fb44387b4542_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-hd7bw_kube-system_c8235f3d-c34b-41b0-afcc-4fdf7ba45905_0 k8s_opensdn-analytics-collector_opensdn-analytics-hd7bw_kube-system_c8235f3d-c34b-41b0-afcc-4fdf7ba45905_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-4tb5j_kube-system_7260b630-b419-45d4-9db7-c014d99cb9de_0 k8s_kafka_opensdn-analytics-alarm-g48fv_kube-system_dd9d4c9c-7556-4228-b225-302c5ec45374_0 k8s_opensdn-controller-config-api_opensdn-controller-config-fc4qg_kube-system_476c310f-5870-4691-b381-1ac8fa51ec8c_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-rdt7g_kube-system_b3cc335e-b7c8-40e5-ab3a-eac5dfa9fdbb_0 k8s_opensdn-controller-control_opensdn-controller-control-5f4z5_kube-system_5bd1373d-fdab-49de-85cf-8b5569f2cfd3_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-jczm4_kube-system_dce1bcc7-4746-4679-8173-306428087321_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-6v6c4_kube-system_07b4d3f6-5895-4537-9573-fb44387b4542_0 k8s_opensdn-analytics-api_opensdn-analytics-hd7bw_kube-system_c8235f3d-c34b-41b0-afcc-4fdf7ba45905_0 k8s_config-zookeeper_config-zookeeper-z85zm_kube-system_0906864b-f64f-4726-b04c-0d737288dd5c_0 k8s_redis_redis-778t5_kube-system_5f2ff040-2688-45c5-ae47-0d8260aa045c_0 k8s_opensdn-agent-nodemgr_opensdn-agent-xfgvz_kube-system_8fec851f-b78a-4259-a9af-6303fb913d89_0 k8s_opensdn-configdb_opensdn-configdb-4tb5j_kube-system_7260b630-b419-45d4-9db7-c014d99cb9de_0 k8s_rabbitmq_rabbitmq-94c4b_kube-system_f857fce5-e2e9-45ba-aa4c-f45bd08eeb82_0 k8s_opensdn-vrouter-agent_opensdn-agent-xfgvz_kube-system_8fec851f-b78a-4259-a9af-6303fb913d89_0 k8s_opensdn-kube-manager_opensdn-kube-manager-8hp9b_kube-system_76e9a8fb-1a9e-4b3b-983e-2beb4a5753e4_0 k8s_kube-proxy_kube-proxy-hl9cz_kube-system_1f3cc471-1b87-4ae7-834e-2dabcf1ed6c9_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-331-1_kube-system_f40a0831a1de574314a6a5f02405a01e_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-331-1_kube-system_631f62ac20b728e0953961d9a99b0d6b_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-331-1_kube-system_ffa908095cdccef932379701a8a94772_0 2026-02-20 12:18:39,927 - DEBUG - [10.0.0.240]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-20 12:18:40,035 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.240/24 2026-02-20 12:18:40,035 - DEBUG - [10.0.0.240]: Running cmd : getent hosts 10.0.0.240 | head -n 1 | awk '{print $2}' 2026-02-20 12:18:40,148 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-331-1.cluster.local 2026-02-20 12:18:40,148 - DEBUG - [10.0.0.240]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-20 12:18:40,263 - DEBUG - Output : 10.0.0.240/24 2026-02-20 12:18:40,612 - INFO - Using existing project ['default-domain', 'k8s-default'](ed7d7364-c78b-4a09-bff8-3bf3094d4a41) 2026-02-20 12:18:40,747 - INFO - Using existing project ['default-domain', 'k8s-default'](ed7d7364-c78b-4a09-bff8-3bf3094d4a41) 2026-02-20 12:18:40,876 - INFO - ================================================================================ 2026-02-20 12:18:40,876 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-20 12:18:40,876 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-20 12:18:41,137 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 12:18:41,142 - DEBUG - Nothing to compare xmpp stats {'10.0.0.240': {'10.0.0.240': '1'}} with 2026-02-20 12:18:41,142 - INFO - Initial checks done. Running the testcase now 2026-02-20 12:18:41,142 - INFO - 2026-02-20 12:18:41,147 - INFO - Namespace default already exists 2026-02-20 12:18:41,147 - DEBUG - [10.0.0.240]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-8hp9b_kube-system_76e9a8fb-1a9e-4b3b-983e-2beb4a5753e4_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-20 12:18:41,324 - DEBUG - Output : cluster_project={} 2026-02-20 12:18:41,397 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-20 12:18:41,397 - DEBUG - Project uuid in Contrail is ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:18:41,397 - DEBUG - [10.0.0.240]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-8hp9b_kube-system_76e9a8fb-1a9e-4b3b-983e-2beb4a5753e4_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-20 12:18:41,535 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-20 12:18:41,535 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_NamespaceDatabaseList?namespace_uuid=ed6b30e3-e2c0-4b49-8e59-a44bdf2317c9 2026-02-20 12:18:41,539 - INFO - Namespace default with uuid ed6b30e3-e2c0-4b49-8e59-a44bdf2317c9 found in kube manager 2026-02-20 12:18:41,539 - INFO - Namespace default verification passed 2026-02-20 12:18:41,543 - DEBUG - Service ctest-nginx-svc-32146693 not present 2026-02-20 12:18:41,543 - INFO - Creating service ctest-nginx-svc-32146693 2026-02-20 12:18:41,568 - INFO - Created Service ctest-nginx-svc-32146693 2026-02-20 12:18:41,573 - DEBUG - Pod ctest-nginx-pod-83735009 not present 2026-02-20 12:18:41,573 - INFO - Creating Pod ctest-nginx-pod-83735009 2026-02-20 12:18:41,606 - DEBUG - Pod : ctest-nginx-pod-83735009 UUID is 5d1cafe0-9989-481f-981f-877058c5293b 2026-02-20 12:18:41,609 - DEBUG - Pod ctest-nginx-pod-31810518 not present 2026-02-20 12:18:41,610 - INFO - Creating Pod ctest-nginx-pod-31810518 2026-02-20 12:18:41,654 - DEBUG - Pod : ctest-nginx-pod-31810518 UUID is 4fab80da-597d-4961-a6ff-0a9021edd1cb 2026-02-20 12:18:41,932 - INFO - Created VN __public__, UUID :5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:42,174 - DEBUG - Requesting: http://10.0.0.240:8082/domains 2026-02-20 12:18:42,179 - DEBUG - Requesting: http://10.0.0.240:8082/domain/93719e63-807c-4adc-9cff-59b2dbd2d4c7 2026-02-20 12:18:42,192 - DEBUG - Requesting: http://10.0.0.240:8082/project/ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:18:42,290 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:42,312 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:42,322 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:42,329 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:42,334 - DEBUG - Route Targets: [] 2026-02-20 12:18:42,334 - DEBUG - RT names not yet present for VN __public__ 2026-02-20 12:18:47,335 - DEBUG - Requesting: http://10.0.0.240:8082/domains 2026-02-20 12:18:47,340 - DEBUG - Requesting: http://10.0.0.240:8082/domain/93719e63-807c-4adc-9cff-59b2dbd2d4c7 2026-02-20 12:18:47,347 - DEBUG - Requesting: http://10.0.0.240:8082/project/ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:18:47,389 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:47,396 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:47,404 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:47,410 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:47,416 - DEBUG - Requesting: http://10.0.0.240:8082/route-target/263680fa-60a7-4223-b1ce-8ba9302d739f 2026-02-20 12:18:47,421 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-20 12:18:47,422 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:47,428 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:47,440 - INFO - Verified VN network id 11 for VN 5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:47,440 - INFO - Verifications in API Server for VN __public__ passed 2026-02-20 12:18:47,440 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:47,448 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:47,453 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:18:47,458 - DEBUG - Requesting: http://10.0.0.240:8082/route-target/263680fa-60a7-4223-b1ce-8ba9302d739f 2026-02-20 12:18:47,463 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-20 12:18:47,470 - DEBUG - Control-node 10.0.0.240 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '6163368562955537332', 'uuid-lslong': '12080453311864562685'}, 'enable': 'true', 'created': '2026-02-20T12:18:41', 'last-modified': '2026-02-20T12:18:42', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.409958'} 2026-02-20 12:18:47,470 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:18:47,478 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-20 12:18:47,479 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-20 12:18:47,479 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-20 12:18:47,479 - DEBUG - Requesting: http://10.0.0.240:8082/domains 2026-02-20 12:18:47,483 - DEBUG - Requesting: http://10.0.0.240:8082/domain/93719e63-807c-4adc-9cff-59b2dbd2d4c7 2026-02-20 12:18:47,490 - DEBUG - Requesting: http://10.0.0.240:8082/project/ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:18:47,522 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:47,530 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-20 12:18:47,530 - DEBUG - Verifying the vn in opserver 2026-02-20 12:18:47,530 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.240 2026-02-20 12:18:47,530 - DEBUG - Requesting: http://10.0.0.240:8081/analytics/uves/virtual-networks 2026-02-20 12:18:47,534 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.240:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-20 12:18:47,534 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-20 12:18:47,534 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:18:47,540 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:18:47,545 - DEBUG - Do not have enough data to verify VN in agent 2026-02-20 12:18:47,545 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:18:47,550 - DEBUG - VRF ids for VN __public__: {} 2026-02-20 12:18:47,707 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-20 12:18:47,906 - DEBUG - Requesting: http://10.0.0.240:8082/domains 2026-02-20 12:18:47,913 - DEBUG - Requesting: http://10.0.0.240:8082/domain/93719e63-807c-4adc-9cff-59b2dbd2d4c7 2026-02-20 12:18:47,927 - DEBUG - Requesting: http://10.0.0.240:8082/project/ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:18:47,992 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:48,004 - DEBUG - Requesting: http://10.0.0.240:8082/floating-ip-pool/eb9c16fc-46e8-45df-92eb-1b2d65ced37f 2026-02-20 12:18:48,014 - DEBUG - Requesting: http://10.0.0.240:8082/domains 2026-02-20 12:18:48,020 - DEBUG - Requesting: http://10.0.0.240:8082/domain/93719e63-807c-4adc-9cff-59b2dbd2d4c7 2026-02-20 12:18:48,036 - DEBUG - Requesting: http://10.0.0.240:8082/project/ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:18:48,093 - DEBUG - Requesting: http://10.0.0.240:8082/virtual-network/5588af0d-96c6-4bb4-a7a6-64483b60e3fd 2026-02-20 12:18:48,136 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-20 12:18:48,137 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-20 12:18:48,149 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 12:18:48,156 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-20 12:18:48,156 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-20 12:18:48,163 - DEBUG - Ingress ctest-nginx-ingress-00414739 not present 2026-02-20 12:18:48,163 - INFO - Creating Ingress ctest-nginx-ingress-00414739 2026-02-20 12:18:48,186 - INFO - Created Ingress ctest-nginx-ingress-00414739 2026-02-20 12:18:48,191 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-00414739 2026-02-20 12:18:51,196 - DEBUG - For Ingress ctest-nginx-ingress-00414739, Cluster IP: 10.47.255.248, External IPs ['11.18.217.124'] 2026-02-20 12:18:51,197 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-00414739 2026-02-20 12:18:51,197 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_LoadbalancerDatabaseList?x=ae015e6c-55a5-46bf-9a17-08294d094461 2026-02-20 12:18:51,202 - INFO - Ingress ctest-nginx-ingress-00414739 with uuid ae015e6c-55a5-46bf-9a17-08294d094461 found in kube manager 2026-02-20 12:18:51,202 - INFO - Ingress ctest-nginx-ingress-00414739 verification passed 2026-02-20 12:18:51,207 - DEBUG - Pod ctest-busybox-pod-56850954 not present 2026-02-20 12:18:51,207 - INFO - Creating Pod ctest-busybox-pod-56850954 2026-02-20 12:18:51,251 - DEBUG - Pod : ctest-busybox-pod-56850954 UUID is fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:18:51,273 - DEBUG - Pod ctest-nginx-pod-83735009 not in running state.Currently in Pending 2026-02-20 12:18:56,282 - INFO - Pod ctest-nginx-pod-83735009 is in running state.Got IP 10.47.255.250 2026-02-20 12:18:56,341 - DEBUG - Pod ctest-nginx-pod-83735009 has vmi 4ca18aa8-0e56-11f1-b0d2-02428fde8ff7 2026-02-20 12:18:56,341 - INFO - Verified pod ctest-nginx-pod-83735009 in contrail-api 2026-02-20 12:18:56,341 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 12:18:56,350 - DEBUG - VMI 4ca18aa8-0e56-11f1-b0d2-02428fde8ff7 is active in agent 10.0.0.240 2026-02-20 12:18:56,351 - INFO - Verified Pod ctest-nginx-pod-83735009 in agent 10.0.0.240 2026-02-20 12:18:56,351 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_VirtualMachineDatabaseList?x=5d1cafe0-9989-481f-981f-877058c5293b 2026-02-20 12:18:56,357 - INFO - Pod ctest-nginx-pod-83735009 with uuid 5d1cafe0-9989-481f-981f-877058c5293b found in kube manager 2026-02-20 12:18:56,357 - INFO - Pod ctest-nginx-pod-83735009 verification passed 2026-02-20 12:18:56,500 - DEBUG - [Pod ctest-nginx-pod-83735009] Cmd: echo ctest-nginx-pod-83735009 > /usr/share/nginx/html/index.html, Output: 2026-02-20 12:18:56,508 - INFO - Pod ctest-nginx-pod-31810518 is in running state.Got IP 10.47.255.249 2026-02-20 12:18:56,589 - DEBUG - Pod ctest-nginx-pod-31810518 has vmi 4d47c6fc-0e56-11f1-b0d2-02428fde8ff7 2026-02-20 12:18:56,589 - INFO - Verified pod ctest-nginx-pod-31810518 in contrail-api 2026-02-20 12:18:56,589 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 12:18:56,601 - DEBUG - VMI 4d47c6fc-0e56-11f1-b0d2-02428fde8ff7 is active in agent 10.0.0.240 2026-02-20 12:18:56,601 - INFO - Verified Pod ctest-nginx-pod-31810518 in agent 10.0.0.240 2026-02-20 12:18:56,602 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_VirtualMachineDatabaseList?x=4fab80da-597d-4961-a6ff-0a9021edd1cb 2026-02-20 12:18:56,614 - INFO - Pod ctest-nginx-pod-31810518 with uuid 4fab80da-597d-4961-a6ff-0a9021edd1cb found in kube manager 2026-02-20 12:18:56,614 - INFO - Pod ctest-nginx-pod-31810518 verification passed 2026-02-20 12:18:56,711 - DEBUG - [Pod ctest-nginx-pod-31810518] Cmd: echo ctest-nginx-pod-31810518 > /usr/share/nginx/html/index.html, Output: 2026-02-20 12:18:56,718 - DEBUG - Pod ctest-busybox-pod-56850954 not in running state.Currently in Pending 2026-02-20 12:19:01,727 - INFO - Pod ctest-busybox-pod-56850954 is in running state.Got IP 10.47.255.247 2026-02-20 12:19:01,794 - DEBUG - Pod ctest-busybox-pod-56850954 has vmi 52aee04e-0e56-11f1-b0d2-02428fde8ff7 2026-02-20 12:19:01,794 - INFO - Verified pod ctest-busybox-pod-56850954 in contrail-api 2026-02-20 12:19:01,794 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 12:19:01,804 - DEBUG - VMI 52aee04e-0e56-11f1-b0d2-02428fde8ff7 is active in agent 10.0.0.240 2026-02-20 12:19:01,804 - INFO - Verified Pod ctest-busybox-pod-56850954 in agent 10.0.0.240 2026-02-20 12:19:01,804 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_VirtualMachineDatabaseList?x=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:01,808 - INFO - Pod ctest-busybox-pod-56850954 with uuid fcf14175-606b-4f6c-84db-24590d0bebf2 found in kube manager 2026-02-20 12:19:01,809 - INFO - Pod ctest-busybox-pod-56850954 verification passed 2026-02-20 12:19:01,883 - DEBUG - [Pod ctest-busybox-pod-56850954] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-83735009 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 12:19:01,883 - DEBUG - [Pod Pod ctest-busybox-pod-56850954] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-20 12:19:01,883 - DEBUG - [Pod Pod ctest-busybox-pod-56850954] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-83735009 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 12:19:01,971 - DEBUG - [Pod ctest-busybox-pod-56850954] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-31810518 2026-02-20 12:19:01,971 - DEBUG - [Pod Pod ctest-busybox-pod-56850954] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-20 12:19:01,972 - DEBUG - [Pod Pod ctest-busybox-pod-56850954] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-31810518 2026-02-20 12:19:01,972 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-83735009': 1, 'ctest-nginx-pod-31810518': 1} 2026-02-20 12:19:01,972 - WARNING - Nothing to delete parallely 2026-02-20 12:19:01,972 - INFO - Deleting pod default:ctest-busybox-pod-56850954 2026-02-20 12:19:01,995 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:02,000 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:07,000 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:07,005 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:12,006 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:12,011 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:17,012 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:17,017 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:22,017 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:22,022 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:27,023 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:27,029 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:32,029 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:32,035 - WARNING - Pod uuid fcf14175-606b-4f6c-84db-24590d0bebf2 is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:37,036 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=fcf14175-606b-4f6c-84db-24590d0bebf2 2026-02-20 12:19:37,041 - DEBUG - Pod fcf14175-606b-4f6c-84db-24590d0bebf2 is not in agent 10.0.0.240 VM list 2026-02-20 12:19:37,041 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:52aee04e-0e56-11f1-b0d2-02428fde8ff7 2026-02-20 12:19:37,045 - DEBUG - VMI 52aee04e-0e56-11f1-b0d2-02428fde8ff7 is removed from agent 10.0.0.240 2026-02-20 12:19:37,045 - INFO - Verified that pod ctest-busybox-pod-56850954 is removed in agent 2026-02-20 12:19:37,045 - INFO - Deleting Ingress : ctest-nginx-ingress-00414739 2026-02-20 12:19:37,051 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_LoadbalancerDatabaseList?x=ae015e6c-55a5-46bf-9a17-08294d094461 2026-02-20 12:19:37,054 - ERROR - Ingress ctest-nginx-ingress-00414739 with uuid ae015e6c-55a5-46bf-9a17-08294d094461 still found in kube manager 2026-02-20 12:19:38,055 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_LoadbalancerDatabaseList?x=ae015e6c-55a5-46bf-9a17-08294d094461 2026-02-20 12:19:38,059 - ERROR - Ingress ctest-nginx-ingress-00414739 with uuid ae015e6c-55a5-46bf-9a17-08294d094461 still found in kube manager 2026-02-20 12:19:39,060 - DEBUG - Requesting: http://10.0.0.240:8108/Snh_LoadbalancerDatabaseList?x=ae015e6c-55a5-46bf-9a17-08294d094461 2026-02-20 12:19:39,064 - INFO - Ingress ctest-nginx-ingress-00414739 with uuid ae015e6c-55a5-46bf-9a17-08294d094461 deleted successfully from kube manager 2026-02-20 12:19:39,064 - INFO - Verifications on Ingress %s cleanup passed 2026-02-20 12:19:39,064 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-20 12:19:39,207 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 12:19:39,223 - WARNING - Control-node ifmap object for FIP pool __fip_pool_public__ , VN __public__ is found! 2026-02-20 12:19:44,224 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 12:19:44,229 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-20 12:19:44,229 - INFO - Deleting VN __public__ 2026-02-20 12:19:44,320 - DEBUG - Requesting: http://10.0.0.240:8082/routing-instance/6ce28acf-076c-4d1f-bc79-1475be753207 2026-02-20 12:19:44,324 - DEBUG - Response Code: 404 2026-02-20 12:19:44,324 - DEBUG - Requesting: http://10.0.0.240:8082/domains 2026-02-20 12:19:44,328 - DEBUG - Requesting: http://10.0.0.240:8082/domain/93719e63-807c-4adc-9cff-59b2dbd2d4c7 2026-02-20 12:19:44,338 - DEBUG - Requesting: http://10.0.0.240:8082/project/ed7d7364-c78b-4a09-bff8-3bf3094d4a41 2026-02-20 12:19:44,380 - INFO - Validated that VN __public__ is not found in API Server 2026-02-20 12:19:44,380 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-20 12:19:44,385 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:19:44,390 - DEBUG - VN __public__ is not present in Agent 10.0.0.240 2026-02-20 12:19:44,390 - INFO - Validated that VN __public__ is not in any agent 2026-02-20 12:19:44,390 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:19:44,395 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 12:19:44,401 - DEBUG - Requesting: http://10.0.0.240:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-20 12:19:44,409 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-20 12:19:44,409 - INFO - Deleting pod default:ctest-nginx-pod-31810518 2026-02-20 12:19:44,420 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=4fab80da-597d-4961-a6ff-0a9021edd1cb 2026-02-20 12:19:44,427 - WARNING - Pod uuid 4fab80da-597d-4961-a6ff-0a9021edd1cb is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:49,428 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=4fab80da-597d-4961-a6ff-0a9021edd1cb 2026-02-20 12:19:49,434 - DEBUG - Pod 4fab80da-597d-4961-a6ff-0a9021edd1cb is not in agent 10.0.0.240 VM list 2026-02-20 12:19:49,434 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4d47c6fc-0e56-11f1-b0d2-02428fde8ff7 2026-02-20 12:19:49,440 - DEBUG - VMI 4d47c6fc-0e56-11f1-b0d2-02428fde8ff7 is removed from agent 10.0.0.240 2026-02-20 12:19:49,440 - INFO - Verified that pod ctest-nginx-pod-31810518 is removed in agent 2026-02-20 12:19:49,440 - INFO - Deleting pod default:ctest-nginx-pod-83735009 2026-02-20 12:19:49,459 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=5d1cafe0-9989-481f-981f-877058c5293b 2026-02-20 12:19:49,467 - WARNING - Pod uuid 5d1cafe0-9989-481f-981f-877058c5293b is still seen in agent 10.0.0.240 VM list 2026-02-20 12:19:54,468 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_VmListReq?uuid=5d1cafe0-9989-481f-981f-877058c5293b 2026-02-20 12:19:54,472 - DEBUG - Pod 5d1cafe0-9989-481f-981f-877058c5293b is not in agent 10.0.0.240 VM list 2026-02-20 12:19:54,472 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4ca18aa8-0e56-11f1-b0d2-02428fde8ff7 2026-02-20 12:19:54,476 - DEBUG - VMI 4ca18aa8-0e56-11f1-b0d2-02428fde8ff7 is removed from agent 10.0.0.240 2026-02-20 12:19:54,476 - INFO - Verified that pod ctest-nginx-pod-83735009 is removed in agent 2026-02-20 12:19:54,476 - INFO - Deleting service : ctest-nginx-svc-32146693 2026-02-20 12:19:55,014 - DEBUG - Requesting: http://10.0.0.240:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 12:19:55,021 - DEBUG - No XMPP flaps were noticed during the test