2026-04-13 01:37:08,681 - DEBUG - [10.0.0.38]: Running cmd : hostname 2026-04-13 01:37:08,967 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-391-1 2026-04-13 01:37:08,967 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2026-04-13 01:37:09,102 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-391-1.cluster.local 2026-04-13 01:37:09,102 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-04-13 01:37:09,325 - DEBUG - Output : NAMES contrail_test_KoKcMqeXy k8s_autoscaler_dns-autoscaler-595558c478-4nbs8_kube-system_e7c3f4a7-8d81-4eae-99bb-b51fb670ccff_0 k8s_coredns_coredns-77f7cc69db-cntct_kube-system_3f7049ff-183d-4aa4-903f-d2b9b8323eaf_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-ww9tj_kube-system_01e2649a-aac8-4a72-8461-9349b5e674a1_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-ww9tj_kube-system_01e2649a-aac8-4a72-8461-9349b5e674a1_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-ww9tj_kube-system_01e2649a-aac8-4a72-8461-9349b5e674a1_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-hkkk9_kube-system_911069c8-28ac-470a-92ef-08b6776d546b_0 k8s_opensdn-controller-control-named_opensdn-controller-control-hkkk9_kube-system_911069c8-28ac-470a-92ef-08b6776d546b_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-l9c6t_kube-system_a04f9d98-aa42-47ee-877b-2fe1b4616ea1_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-drfnw_kube-system_2a29304e-7aa3-44a9-96a8-cc36eac3bd4a_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-ltrnz_kube-system_b005d4fe-b461-4530-982b-9981d745c2b2_0 k8s_opensdn-analytics-collector_opensdn-analytics-ltrnz_kube-system_b005d4fe-b461-4530-982b-9981d745c2b2_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-ww9tj_kube-system_01e2649a-aac8-4a72-8461-9349b5e674a1_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-zx252_kube-system_bc0a5243-e5ef-4857-87a2-4a2fd335691e_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-zx252_kube-system_bc0a5243-e5ef-4857-87a2-4a2fd335691e_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-hkkk9_kube-system_911069c8-28ac-470a-92ef-08b6776d546b_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-b9v2c_kube-system_1cc28700-3da7-4291-8540-32d381c107d0_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-b9v2c_kube-system_1cc28700-3da7-4291-8540-32d381c107d0_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-l9c6t_kube-system_a04f9d98-aa42-47ee-877b-2fe1b4616ea1_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-drfnw_kube-system_2a29304e-7aa3-44a9-96a8-cc36eac3bd4a_0 k8s_opensdn-analytics-api_opensdn-analytics-ltrnz_kube-system_b005d4fe-b461-4530-982b-9981d745c2b2_0 k8s_opensdn-controller-config-api_opensdn-controller-config-ww9tj_kube-system_01e2649a-aac8-4a72-8461-9349b5e674a1_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-zx252_kube-system_bc0a5243-e5ef-4857-87a2-4a2fd335691e_0 k8s_opensdn-controller-control_opensdn-controller-control-hkkk9_kube-system_911069c8-28ac-470a-92ef-08b6776d546b_0 k8s_kafka_opensdn-analytics-alarm-b9v2c_kube-system_1cc28700-3da7-4291-8540-32d381c107d0_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-ghbdx_kube-system_70962fcc-100e-4d26-97b3-1fdbfe5307ba_0 k8s_opensdn-configdb_opensdn-configdb-ghbdx_kube-system_70962fcc-100e-4d26-97b3-1fdbfe5307ba_0 k8s_config-zookeeper_config-zookeeper-wbk5z_kube-system_a805ef95-f57f-467b-9a80-a010ac14abf4_0 k8s_redis_redis-hfh6x_kube-system_c8122c98-df58-4305-875e-55919e8d8133_0 k8s_rabbitmq_rabbitmq-n24dh_kube-system_c44e34bd-f348-4555-beaf-396d2f719a3c_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-l9c6t_kube-system_a04f9d98-aa42-47ee-877b-2fe1b4616ea1_0 k8s_opensdn-agent-nodemgr_opensdn-agent-bgr4h_kube-system_2298221c-40f8-4705-bdc0-2339a6ed3228_0 k8s_opensdn-vrouter-agent_opensdn-agent-bgr4h_kube-system_2298221c-40f8-4705-bdc0-2339a6ed3228_0 k8s_opensdn-kube-manager_opensdn-kube-manager-xs6r5_kube-system_9982dd8b-2a8f-403c-9803-7acf58053c7b_0 k8s_kube-proxy_kube-proxy-rt5bm_kube-system_fb07b2fd-14dc-4661-ad00-b6cb54f98e12_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-391-1_kube-system_9d578eed0e01fc57c7140577daef3880_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-391-1_kube-system_79c6b057aeea94b7bd8f6debdecc7cf9_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-391-1_kube-system_47869240efbcf7804d40724731fd528d_0 2026-04-13 01:37:09,326 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-04-13 01:37:09,466 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.1/32 10.96.0.3/32 10.0.0.38/24 2026-04-13 01:37:09,466 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2026-04-13 01:37:09,602 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-391-1.cluster.local 2026-04-13 01:37:09,603 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-04-13 01:37:09,730 - DEBUG - Output : 10.0.0.38/24 2026-04-13 01:37:10,145 - INFO - Using existing project ['default-domain', 'k8s-default'](0f9e6ea6-e5c0-4c1b-8444-737beedfee52) 2026-04-13 01:37:10,339 - INFO - Using existing project ['default-domain', 'k8s-default'](0f9e6ea6-e5c0-4c1b-8444-737beedfee52) 2026-04-13 01:37:10,469 - INFO - ================================================================================ 2026-04-13 01:37:10,469 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-04-13 01:37:10,470 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-04-13 01:37:10,782 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-04-13 01:37:10,788 - DEBUG - Nothing to compare xmpp stats {'10.0.0.38': {'10.0.0.38': '2'}} with 2026-04-13 01:37:10,788 - INFO - Initial checks done. Running the testcase now 2026-04-13 01:37:10,789 - INFO - 2026-04-13 01:37:10,793 - INFO - Namespace default already exists 2026-04-13 01:37:10,794 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-xs6r5_kube-system_9982dd8b-2a8f-403c-9803-7acf58053c7b_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-04-13 01:37:10,980 - DEBUG - Output : cluster_project={} 2026-04-13 01:37:11,060 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-04-13 01:37:11,061 - DEBUG - Project uuid in Contrail is 0f9e6ea6-e5c0-4c1b-8444-737beedfee52 2026-04-13 01:37:11,061 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-xs6r5_kube-system_9982dd8b-2a8f-403c-9803-7acf58053c7b_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-04-13 01:37:11,230 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-04-13 01:37:11,230 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_NamespaceDatabaseList?namespace_uuid=25d61374-c273-4557-a3b9-5b1a06860959 2026-04-13 01:37:11,235 - INFO - Namespace default with uuid 25d61374-c273-4557-a3b9-5b1a06860959 found in kube manager 2026-04-13 01:37:11,235 - INFO - Namespace default verification passed 2026-04-13 01:37:11,238 - DEBUG - Service ctest-nginx-svc-01270822 not present 2026-04-13 01:37:11,239 - INFO - Creating service ctest-nginx-svc-01270822 2026-04-13 01:37:11,256 - INFO - Created Service ctest-nginx-svc-01270822 2026-04-13 01:37:11,265 - DEBUG - Pod ctest-nginx-pod-30189663 not present 2026-04-13 01:37:11,266 - INFO - Creating Pod ctest-nginx-pod-30189663 2026-04-13 01:37:11,301 - DEBUG - Pod : ctest-nginx-pod-30189663 UUID is 145c9500-3d89-4fbf-9e05-9c521d530912 2026-04-13 01:37:11,307 - DEBUG - Pod ctest-nginx-pod-14808707 not present 2026-04-13 01:37:11,307 - INFO - Creating Pod ctest-nginx-pod-14808707 2026-04-13 01:37:11,353 - DEBUG - Pod : ctest-nginx-pod-14808707 UUID is d38b215b-69e7-408e-9b86-158ce760839d 2026-04-13 01:37:12,333 - INFO - Created VN __public__, UUID :cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:12,909 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-04-13 01:37:12,916 - DEBUG - Requesting: http://10.0.0.38:8082/domain/b5c76fa4-125b-4a3d-80e9-afedf08fc6bf 2026-04-13 01:37:12,944 - DEBUG - Requesting: http://10.0.0.38:8082/project/0f9e6ea6-e5c0-4c1b-8444-737beedfee52 2026-04-13 01:37:13,053 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,065 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,073 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/a3e3e130-8d39-4149-ad72-e84bf46430b9 2026-04-13 01:37:13,079 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/a3e3e130-8d39-4149-ad72-e84bf46430b9 2026-04-13 01:37:13,085 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/b8082589-b264-4b84-920a-79810dd3a3bb 2026-04-13 01:37:13,095 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-04-13 01:37:13,095 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,103 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/a3e3e130-8d39-4149-ad72-e84bf46430b9 2026-04-13 01:37:13,124 - INFO - Verified VN network id 11 for VN cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,124 - INFO - Verifications in API Server for VN __public__ passed 2026-04-13 01:37:13,124 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,158 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/a3e3e130-8d39-4149-ad72-e84bf46430b9 2026-04-13 01:37:13,174 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/a3e3e130-8d39-4149-ad72-e84bf46430b9 2026-04-13 01:37:13,182 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/b8082589-b264-4b84-920a-79810dd3a3bb 2026-04-13 01:37:13,189 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-04-13 01:37:13,196 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '14947038599451069611', 'uuid-lslong': '12231416654204455406'}, 'enable': 'true', 'created': '2026-04-13T01:37:12', 'last-modified': '2026-04-13T01:37:12', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.548392'} 2026-04-13 01:37:13,196 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:37:13,205 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-04-13 01:37:13,205 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-04-13 01:37:13,206 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-04-13 01:37:13,206 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-04-13 01:37:13,214 - DEBUG - Requesting: http://10.0.0.38:8082/domain/b5c76fa4-125b-4a3d-80e9-afedf08fc6bf 2026-04-13 01:37:13,221 - DEBUG - Requesting: http://10.0.0.38:8082/project/0f9e6ea6-e5c0-4c1b-8444-737beedfee52 2026-04-13 01:37:13,253 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,262 - DEBUG - =>VN __public__ has no policy to be verified 2026-04-13 01:37:13,262 - DEBUG - Verifying the vn in opserver 2026-04-13 01:37:13,262 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.38 2026-04-13 01:37:13,262 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2026-04-13 01:37:13,267 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-04-13 01:37:13,267 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-04-13 01:37:13,267 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:37:13,274 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:37:13,280 - DEBUG - Do not have enough data to verify VN in agent 2026-04-13 01:37:13,280 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:37:13,285 - DEBUG - VRF ids for VN __public__: {} 2026-04-13 01:37:13,472 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-04-13 01:37:13,735 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-04-13 01:37:13,743 - DEBUG - Requesting: http://10.0.0.38:8082/domain/b5c76fa4-125b-4a3d-80e9-afedf08fc6bf 2026-04-13 01:37:13,752 - DEBUG - Requesting: http://10.0.0.38:8082/project/0f9e6ea6-e5c0-4c1b-8444-737beedfee52 2026-04-13 01:37:13,788 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,800 - DEBUG - Requesting: http://10.0.0.38:8082/floating-ip-pool/1fc54ff0-2e77-412e-af4a-e21bc633f3e8 2026-04-13 01:37:13,806 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-04-13 01:37:13,817 - DEBUG - Requesting: http://10.0.0.38:8082/domain/b5c76fa4-125b-4a3d-80e9-afedf08fc6bf 2026-04-13 01:37:13,824 - DEBUG - Requesting: http://10.0.0.38:8082/project/0f9e6ea6-e5c0-4c1b-8444-737beedfee52 2026-04-13 01:37:13,855 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/cf6e8c69-d455-4cab-a9be-b8a465a749ee 2026-04-13 01:37:13,863 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-04-13 01:37:13,863 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-04-13 01:37:13,879 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-04-13 01:37:13,885 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-04-13 01:37:13,885 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-04-13 01:37:13,888 - DEBUG - Ingress ctest-nginx-ingress-98557849 not present 2026-04-13 01:37:13,889 - INFO - Creating Ingress ctest-nginx-ingress-98557849 2026-04-13 01:37:13,897 - INFO - Created Ingress ctest-nginx-ingress-98557849 2026-04-13 01:37:13,901 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98557849 2026-04-13 01:37:16,908 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-98557849 2026-04-13 01:37:19,917 - DEBUG - For Ingress ctest-nginx-ingress-98557849, Cluster IP: 10.47.255.248, External IPs ['7.77.33.124'] 2026-04-13 01:37:19,917 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-98557849 2026-04-13 01:37:19,917 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=53e37193-b5d1-41ed-b111-891e87e58088 2026-04-13 01:37:19,934 - INFO - Ingress ctest-nginx-ingress-98557849 with uuid 53e37193-b5d1-41ed-b111-891e87e58088 found in kube manager 2026-04-13 01:37:19,934 - INFO - Ingress ctest-nginx-ingress-98557849 verification passed 2026-04-13 01:37:19,938 - DEBUG - Pod ctest-busybox-pod-97131910 not present 2026-04-13 01:37:19,938 - INFO - Creating Pod ctest-busybox-pod-97131910 2026-04-13 01:37:19,993 - DEBUG - Pod : ctest-busybox-pod-97131910 UUID is 93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:20,009 - DEBUG - Pod ctest-nginx-pod-30189663 not in running state.Currently in Pending 2026-04-13 01:37:25,019 - DEBUG - Pod ctest-nginx-pod-30189663 not in running state.Currently in Pending 2026-04-13 01:37:30,028 - INFO - Pod ctest-nginx-pod-30189663 is in running state.Got IP 10.47.255.250 2026-04-13 01:37:30,083 - DEBUG - Pod ctest-nginx-pod-30189663 has vmi 4c70fc20-36d9-11f1-92fb-024254d8960c 2026-04-13 01:37:30,083 - INFO - Verified pod ctest-nginx-pod-30189663 in contrail-api 2026-04-13 01:37:30,083 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-04-13 01:37:30,096 - DEBUG - VMI 4c70fc20-36d9-11f1-92fb-024254d8960c is active in agent 10.0.0.38 2026-04-13 01:37:30,096 - INFO - Verified Pod ctest-nginx-pod-30189663 in agent 10.0.0.38 2026-04-13 01:37:30,096 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=145c9500-3d89-4fbf-9e05-9c521d530912 2026-04-13 01:37:30,100 - INFO - Pod ctest-nginx-pod-30189663 with uuid 145c9500-3d89-4fbf-9e05-9c521d530912 found in kube manager 2026-04-13 01:37:30,101 - INFO - Pod ctest-nginx-pod-30189663 verification passed 2026-04-13 01:37:30,213 - DEBUG - [Pod ctest-nginx-pod-30189663] Cmd: echo ctest-nginx-pod-30189663 > /usr/share/nginx/html/index.html, Output: 2026-04-13 01:37:30,221 - INFO - Pod ctest-nginx-pod-14808707 is in running state.Got IP 10.47.255.249 2026-04-13 01:37:30,275 - DEBUG - Pod ctest-nginx-pod-14808707 has vmi 4d2c437c-36d9-11f1-92fb-024254d8960c 2026-04-13 01:37:30,275 - INFO - Verified pod ctest-nginx-pod-14808707 in contrail-api 2026-04-13 01:37:30,275 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-04-13 01:37:30,289 - DEBUG - VMI 4d2c437c-36d9-11f1-92fb-024254d8960c is active in agent 10.0.0.38 2026-04-13 01:37:30,289 - INFO - Verified Pod ctest-nginx-pod-14808707 in agent 10.0.0.38 2026-04-13 01:37:30,290 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=d38b215b-69e7-408e-9b86-158ce760839d 2026-04-13 01:37:30,294 - INFO - Pod ctest-nginx-pod-14808707 with uuid d38b215b-69e7-408e-9b86-158ce760839d found in kube manager 2026-04-13 01:37:30,295 - INFO - Pod ctest-nginx-pod-14808707 verification passed 2026-04-13 01:37:30,369 - DEBUG - [Pod ctest-nginx-pod-14808707] Cmd: echo ctest-nginx-pod-14808707 > /usr/share/nginx/html/index.html, Output: 2026-04-13 01:37:30,383 - DEBUG - Pod ctest-busybox-pod-97131910 not in running state.Currently in Pending 2026-04-13 01:37:35,391 - INFO - Pod ctest-busybox-pod-97131910 is in running state.Got IP 10.47.255.247 2026-04-13 01:37:35,447 - DEBUG - Pod ctest-busybox-pod-97131910 has vmi 51ba5848-36d9-11f1-92fb-024254d8960c 2026-04-13 01:37:35,447 - INFO - Verified pod ctest-busybox-pod-97131910 in contrail-api 2026-04-13 01:37:35,447 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-04-13 01:37:35,456 - DEBUG - VMI 51ba5848-36d9-11f1-92fb-024254d8960c is active in agent 10.0.0.38 2026-04-13 01:37:35,456 - INFO - Verified Pod ctest-busybox-pod-97131910 in agent 10.0.0.38 2026-04-13 01:37:35,457 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:35,462 - INFO - Pod ctest-busybox-pod-97131910 with uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 found in kube manager 2026-04-13 01:37:35,462 - INFO - Pod ctest-busybox-pod-97131910 verification passed 2026-04-13 01:37:35,567 - DEBUG - [Pod ctest-busybox-pod-97131910] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-14808707 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-04-13 01:37:35,567 - DEBUG - [Pod Pod ctest-busybox-pod-97131910] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-04-13 01:37:35,567 - DEBUG - [Pod Pod ctest-busybox-pod-97131910] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-14808707 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-04-13 01:37:35,661 - DEBUG - [Pod ctest-busybox-pod-97131910] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-30189663 2026-04-13 01:37:35,661 - DEBUG - [Pod Pod ctest-busybox-pod-97131910] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-04-13 01:37:35,661 - DEBUG - [Pod Pod ctest-busybox-pod-97131910] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-30189663 2026-04-13 01:37:35,661 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-30189663': 1, 'ctest-nginx-pod-14808707': 1} 2026-04-13 01:37:35,661 - WARNING - Nothing to delete parallely 2026-04-13 01:37:35,662 - INFO - Deleting pod default:ctest-busybox-pod-97131910 2026-04-13 01:37:35,677 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:35,685 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:37:40,686 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:40,693 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:37:45,694 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:45,700 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:37:50,700 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:50,705 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:37:55,707 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:37:55,716 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:38:00,717 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:38:00,722 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:38:05,724 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:38:05,729 - WARNING - Pod uuid 93b7ed12-f194-46e5-a970-89d938b00eb1 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:38:10,730 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=93b7ed12-f194-46e5-a970-89d938b00eb1 2026-04-13 01:38:10,735 - DEBUG - Pod 93b7ed12-f194-46e5-a970-89d938b00eb1 is not in agent 10.0.0.38 VM list 2026-04-13 01:38:10,735 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:51ba5848-36d9-11f1-92fb-024254d8960c 2026-04-13 01:38:10,740 - DEBUG - VMI 51ba5848-36d9-11f1-92fb-024254d8960c is removed from agent 10.0.0.38 2026-04-13 01:38:10,740 - INFO - Verified that pod ctest-busybox-pod-97131910 is removed in agent 2026-04-13 01:38:10,740 - INFO - Deleting Ingress : ctest-nginx-ingress-98557849 2026-04-13 01:38:10,748 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=53e37193-b5d1-41ed-b111-891e87e58088 2026-04-13 01:38:10,755 - ERROR - Ingress ctest-nginx-ingress-98557849 with uuid 53e37193-b5d1-41ed-b111-891e87e58088 still found in kube manager 2026-04-13 01:38:11,756 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=53e37193-b5d1-41ed-b111-891e87e58088 2026-04-13 01:38:11,770 - ERROR - Ingress ctest-nginx-ingress-98557849 with uuid 53e37193-b5d1-41ed-b111-891e87e58088 still found in kube manager 2026-04-13 01:38:12,772 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=53e37193-b5d1-41ed-b111-891e87e58088 2026-04-13 01:38:12,778 - INFO - Ingress ctest-nginx-ingress-98557849 with uuid 53e37193-b5d1-41ed-b111-891e87e58088 deleted successfully from kube manager 2026-04-13 01:38:12,778 - INFO - Verifications on Ingress %s cleanup passed 2026-04-13 01:38:12,778 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-04-13 01:38:12,909 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-04-13 01:38:12,919 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-04-13 01:38:12,919 - INFO - Deleting VN __public__ 2026-04-13 01:38:13,084 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/a3e3e130-8d39-4149-ad72-e84bf46430b9 2026-04-13 01:38:13,089 - DEBUG - Response Code: 404 2026-04-13 01:38:13,090 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2026-04-13 01:38:13,097 - DEBUG - Requesting: http://10.0.0.38:8082/domain/b5c76fa4-125b-4a3d-80e9-afedf08fc6bf 2026-04-13 01:38:13,107 - DEBUG - Requesting: http://10.0.0.38:8082/project/0f9e6ea6-e5c0-4c1b-8444-737beedfee52 2026-04-13 01:38:13,155 - INFO - Validated that VN __public__ is not found in API Server 2026-04-13 01:38:13,155 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-04-13 01:38:13,160 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:38:13,167 - DEBUG - VN __public__ is not present in Agent 10.0.0.38 2026-04-13 01:38:13,167 - INFO - Validated that VN __public__ is not in any agent 2026-04-13 01:38:13,167 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:38:13,174 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-04-13 01:38:13,203 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-04-13 01:38:13,208 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-04-13 01:38:13,209 - INFO - Deleting pod default:ctest-nginx-pod-14808707 2026-04-13 01:38:13,221 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=d38b215b-69e7-408e-9b86-158ce760839d 2026-04-13 01:38:13,230 - WARNING - Pod uuid d38b215b-69e7-408e-9b86-158ce760839d is still seen in agent 10.0.0.38 VM list 2026-04-13 01:38:18,232 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=d38b215b-69e7-408e-9b86-158ce760839d 2026-04-13 01:38:18,239 - DEBUG - Pod d38b215b-69e7-408e-9b86-158ce760839d is not in agent 10.0.0.38 VM list 2026-04-13 01:38:18,239 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4d2c437c-36d9-11f1-92fb-024254d8960c 2026-04-13 01:38:18,245 - DEBUG - VMI 4d2c437c-36d9-11f1-92fb-024254d8960c is removed from agent 10.0.0.38 2026-04-13 01:38:18,245 - INFO - Verified that pod ctest-nginx-pod-14808707 is removed in agent 2026-04-13 01:38:18,245 - INFO - Deleting pod default:ctest-nginx-pod-30189663 2026-04-13 01:38:18,269 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=145c9500-3d89-4fbf-9e05-9c521d530912 2026-04-13 01:38:18,277 - WARNING - Pod uuid 145c9500-3d89-4fbf-9e05-9c521d530912 is still seen in agent 10.0.0.38 VM list 2026-04-13 01:38:23,278 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_VmListReq?uuid=145c9500-3d89-4fbf-9e05-9c521d530912 2026-04-13 01:38:23,284 - DEBUG - Pod 145c9500-3d89-4fbf-9e05-9c521d530912 is not in agent 10.0.0.38 VM list 2026-04-13 01:38:23,284 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:4c70fc20-36d9-11f1-92fb-024254d8960c 2026-04-13 01:38:23,290 - DEBUG - VMI 4c70fc20-36d9-11f1-92fb-024254d8960c is removed from agent 10.0.0.38 2026-04-13 01:38:23,290 - INFO - Verified that pod ctest-nginx-pod-30189663 is removed in agent 2026-04-13 01:38:23,290 - INFO - Deleting service : ctest-nginx-svc-01270822 2026-04-13 01:38:23,880 - DEBUG - Requesting: http://10.0.0.38:8085/Snh_AgentXmppConnectionStatusReq? 2026-04-13 01:38:23,885 - DEBUG - No XMPP flaps were noticed during the test