2026-02-20 08:44:59,211 - DEBUG - [10.0.0.41]: Running cmd : hostname 2026-02-20 08:44:59,481 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-330-1 2026-02-20 08:44:59,481 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2026-02-20 08:44:59,625 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-330-1.cluster.local 2026-02-20 08:44:59,626 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-02-20 08:44:59,818 - DEBUG - Output : NAMES contrail_test_cdzUyln9O k8s_autoscaler_dns-autoscaler-595558c478-fnlzd_kube-system_8509aa0c-d962-4a52-832e-a4a2a8453c6c_0 k8s_coredns_coredns-77f7cc69db-gm5z8_kube-system_b5ef86ee-b033-41b7-81fa-bc31907139cd_0 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-jmgqf_kube-system_fd207a55-ef09-4f6f-979a-7511e39b0a07_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-jmgqf_kube-system_fd207a55-ef09-4f6f-979a-7511e39b0a07_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-hxthx_kube-system_3803d6e4-d7bc-488a-9500-dc220f0b232a_0 k8s_opensdn-controller-control-named_opensdn-controller-control-hxthx_kube-system_3803d6e4-d7bc-488a-9500-dc220f0b232a_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-jmgqf_kube-system_fd207a55-ef09-4f6f-979a-7511e39b0a07_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-6xdkw_kube-system_9bbdbd9b-3720-4ef4-beb0-19bc204b03d3_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-hxthx_kube-system_3803d6e4-d7bc-488a-9500-dc220f0b232a_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-htkbd_kube-system_a30fdb56-0518-461d-adc5-fa52c77bd1b4_0 k8s_opensdn-analytics-collector_opensdn-analytics-htkbd_kube-system_a30fdb56-0518-461d-adc5-fa52c77bd1b4_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-p7wfc_kube-system_f4970dfc-c1eb-4035-98ad-96447d2e5f02_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-jmgqf_kube-system_fd207a55-ef09-4f6f-979a-7511e39b0a07_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-6xdkw_kube-system_9bbdbd9b-3720-4ef4-beb0-19bc204b03d3_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-zlpsl_kube-system_52de5398-e63c-4c3b-acd9-22b90060ff5c_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-zlpsl_kube-system_52de5398-e63c-4c3b-acd9-22b90060ff5c_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-6tvlp_kube-system_544015bf-93f0-4292-b86a-302e513fabe2_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-6tvlp_kube-system_544015bf-93f0-4292-b86a-302e513fabe2_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-62dct_kube-system_60809b73-ac29-4eb3-a884-10a6cefe463d_0 k8s_opensdn-controller-control_opensdn-controller-control-hxthx_kube-system_3803d6e4-d7bc-488a-9500-dc220f0b232a_0 k8s_opensdn-analytics-api_opensdn-analytics-htkbd_kube-system_a30fdb56-0518-461d-adc5-fa52c77bd1b4_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-p7wfc_kube-system_f4970dfc-c1eb-4035-98ad-96447d2e5f02_0 k8s_opensdn-controller-config-api_opensdn-controller-config-jmgqf_kube-system_fd207a55-ef09-4f6f-979a-7511e39b0a07_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-6xdkw_kube-system_9bbdbd9b-3720-4ef4-beb0-19bc204b03d3_0 k8s_opensdn-agent-nodemgr_opensdn-agent-g6k6g_kube-system_78419c08-0860-43ea-932f-222696e8f760_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-zlpsl_kube-system_52de5398-e63c-4c3b-acd9-22b90060ff5c_0 k8s_kafka_opensdn-analytics-alarm-6tvlp_kube-system_544015bf-93f0-4292-b86a-302e513fabe2_0 k8s_opensdn-configdb_opensdn-configdb-62dct_kube-system_60809b73-ac29-4eb3-a884-10a6cefe463d_0 k8s_rabbitmq_rabbitmq-pz87r_kube-system_6fb694e0-e5f0-4243-8554-a58e7d681aaa_0 k8s_config-zookeeper_config-zookeeper-fsmvh_kube-system_154c002d-3f62-400e-85bc-40a00a5ac467_0 k8s_redis_redis-zb5qr_kube-system_ca920367-6850-4f9b-95be-fe789a8d4378_0 k8s_opensdn-vrouter-agent_opensdn-agent-g6k6g_kube-system_78419c08-0860-43ea-932f-222696e8f760_0 k8s_opensdn-kube-manager_opensdn-kube-manager-7l69l_kube-system_b96a3a19-8b3b-4e89-b9c8-ef11120cea67_0 k8s_kube-proxy_kube-proxy-z9mln_kube-system_2a5c0004-7f95-413e-abb2-11c08fcc565f_0 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-330-1_kube-system_5b344277c5f27d4c02720435a154d4da_1 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-330-1_kube-system_ce791b3873004fe4e8cb4a2dafcd4e09_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-330-1_kube-system_cfcc62442f6d433c9e07d2f2e8097f47_0 2026-02-20 08:44:59,818 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-02-20 08:44:59,946 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.41/24 2026-02-20 08:44:59,946 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.0.0.41 | head -n 1 | awk '{print $2}' 2026-02-20 08:45:00,062 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-330-1.cluster.local 2026-02-20 08:45:00,063 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-02-20 08:45:00,205 - DEBUG - Output : 10.0.0.41/24 2026-02-20 08:45:00,559 - INFO - Using existing project ['default-domain', 'k8s-default'](84460113-fc78-43e1-a90b-4ea2c8c6a334) 2026-02-20 08:45:00,701 - INFO - Using existing project ['default-domain', 'k8s-default'](84460113-fc78-43e1-a90b-4ea2c8c6a334) 2026-02-20 08:45:00,898 - INFO - ================================================================================ 2026-02-20 08:45:00,898 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-02-20 08:45:00,898 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-02-20 08:45:01,212 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 08:45:01,217 - DEBUG - Nothing to compare xmpp stats {'10.0.0.41': {'10.0.0.41': '2'}} with 2026-02-20 08:45:01,218 - INFO - Initial checks done. Running the testcase now 2026-02-20 08:45:01,218 - INFO - 2026-02-20 08:45:01,223 - INFO - Namespace default already exists 2026-02-20 08:45:01,224 - DEBUG - [10.0.0.41]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-7l69l_kube-system_b96a3a19-8b3b-4e89-b9c8-ef11120cea67_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-02-20 08:45:01,411 - DEBUG - Output : cluster_project={} 2026-02-20 08:45:01,498 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-02-20 08:45:01,498 - DEBUG - Project uuid in Contrail is 84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:45:01,498 - DEBUG - [10.0.0.41]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-7l69l_kube-system_b96a3a19-8b3b-4e89-b9c8-ef11120cea67_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-02-20 08:45:01,657 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-02-20 08:45:01,657 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_NamespaceDatabaseList?namespace_uuid=abf0221e-d589-4e23-8120-d954ffca2d86 2026-02-20 08:45:01,661 - INFO - Namespace default with uuid abf0221e-d589-4e23-8120-d954ffca2d86 found in kube manager 2026-02-20 08:45:01,661 - INFO - Namespace default verification passed 2026-02-20 08:45:01,665 - DEBUG - Service ctest-nginx-svc-44847957 not present 2026-02-20 08:45:01,666 - INFO - Creating service ctest-nginx-svc-44847957 2026-02-20 08:45:01,684 - INFO - Created Service ctest-nginx-svc-44847957 2026-02-20 08:45:01,690 - DEBUG - Pod ctest-nginx-pod-18170248 not present 2026-02-20 08:45:01,691 - INFO - Creating Pod ctest-nginx-pod-18170248 2026-02-20 08:45:01,723 - DEBUG - Pod : ctest-nginx-pod-18170248 UUID is ce441269-de00-4ac0-96e8-1c3382ef9626 2026-02-20 08:45:01,727 - DEBUG - Pod ctest-nginx-pod-87203779 not present 2026-02-20 08:45:01,727 - INFO - Creating Pod ctest-nginx-pod-87203779 2026-02-20 08:45:01,751 - DEBUG - Pod : ctest-nginx-pod-87203779 UUID is d08b8776-e39d-4283-bc4d-c7cea36d8b69 2026-02-20 08:45:02,511 - INFO - Created VN __public__, UUID :4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:02,860 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2026-02-20 08:45:02,866 - DEBUG - Requesting: http://10.0.0.41:8082/domain/65444de8-4e2f-4931-97af-17702a2399e5 2026-02-20 08:45:02,879 - DEBUG - Requesting: http://10.0.0.41:8082/project/84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:45:03,002 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:03,019 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:03,028 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:03,039 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:03,051 - DEBUG - Route Targets: [] 2026-02-20 08:45:03,051 - DEBUG - RT names not yet present for VN __public__ 2026-02-20 08:45:08,052 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2026-02-20 08:45:08,057 - DEBUG - Requesting: http://10.0.0.41:8082/domain/65444de8-4e2f-4931-97af-17702a2399e5 2026-02-20 08:45:08,063 - DEBUG - Requesting: http://10.0.0.41:8082/project/84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:45:08,099 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,108 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,117 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:08,124 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:08,129 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/ac7bcc2e-3543-4c00-92eb-af62dde49f1e 2026-02-20 08:45:08,134 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-20 08:45:08,134 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,141 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:08,161 - INFO - Verified VN network id 11 for VN 4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,161 - INFO - Verifications in API Server for VN __public__ passed 2026-02-20 08:45:08,161 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,174 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:08,181 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:45:08,187 - DEBUG - Requesting: http://10.0.0.41:8082/route-target/ac7bcc2e-3543-4c00-92eb-af62dde49f1e 2026-02-20 08:45:08,192 - DEBUG - Requesting: http://10.0.0.41:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-20 08:45:08,201 - DEBUG - Control-node 10.0.0.41 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5596478138092046607', 'uuid-lslong': '11837704068848891192'}, 'enable': 'true', 'created': '2026-02-20T08:45:02', 'last-modified': '2026-02-20T08:45:02', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.472105'} 2026-02-20 08:45:08,201 - DEBUG - Requesting: http://10.0.0.41:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:45:08,209 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-02-20 08:45:08,209 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-02-20 08:45:08,210 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-02-20 08:45:08,210 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2026-02-20 08:45:08,214 - DEBUG - Requesting: http://10.0.0.41:8082/domain/65444de8-4e2f-4931-97af-17702a2399e5 2026-02-20 08:45:08,220 - DEBUG - Requesting: http://10.0.0.41:8082/project/84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:45:08,240 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,256 - DEBUG - =>VN __public__ has no policy to be verified 2026-02-20 08:45:08,256 - DEBUG - Verifying the vn in opserver 2026-02-20 08:45:08,256 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.41 2026-02-20 08:45:08,256 - DEBUG - Requesting: http://10.0.0.41:8081/analytics/uves/virtual-networks 2026-02-20 08:45:08,261 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.41:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-02-20 08:45:08,261 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-02-20 08:45:08,261 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:45:08,268 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:45:08,273 - DEBUG - Do not have enough data to verify VN in agent 2026-02-20 08:45:08,273 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:45:08,277 - DEBUG - VRF ids for VN __public__: {} 2026-02-20 08:45:08,443 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-02-20 08:45:08,689 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2026-02-20 08:45:08,700 - DEBUG - Requesting: http://10.0.0.41:8082/domain/65444de8-4e2f-4931-97af-17702a2399e5 2026-02-20 08:45:08,707 - DEBUG - Requesting: http://10.0.0.41:8082/project/84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:45:08,764 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,789 - DEBUG - Requesting: http://10.0.0.41:8082/floating-ip-pool/d4e5d9e8-1cd6-4065-ad59-4ff7c21b5f9d 2026-02-20 08:45:08,795 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2026-02-20 08:45:08,799 - DEBUG - Requesting: http://10.0.0.41:8082/domain/65444de8-4e2f-4931-97af-17702a2399e5 2026-02-20 08:45:08,806 - DEBUG - Requesting: http://10.0.0.41:8082/project/84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:45:08,857 - DEBUG - Requesting: http://10.0.0.41:8082/virtual-network/4daaaf37-1637-490f-a447-f921bbef3538 2026-02-20 08:45:08,865 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-02-20 08:45:08,865 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-02-20 08:45:08,871 - DEBUG - Requesting: http://10.0.0.41:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 08:45:08,877 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-02-20 08:45:08,877 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-02-20 08:45:08,880 - DEBUG - Ingress ctest-nginx-ingress-75780102 not present 2026-02-20 08:45:08,881 - INFO - Creating Ingress ctest-nginx-ingress-75780102 2026-02-20 08:45:08,887 - INFO - Created Ingress ctest-nginx-ingress-75780102 2026-02-20 08:45:08,892 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-75780102 2026-02-20 08:45:11,898 - DEBUG - For Ingress ctest-nginx-ingress-75780102, Cluster IP: 10.47.255.248, External IPs ['91.120.211.252'] 2026-02-20 08:45:11,898 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-75780102 2026-02-20 08:45:11,898 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_LoadbalancerDatabaseList?x=c186598a-3c83-43a5-ab23-c1b9704818fe 2026-02-20 08:45:11,906 - INFO - Ingress ctest-nginx-ingress-75780102 with uuid c186598a-3c83-43a5-ab23-c1b9704818fe found in kube manager 2026-02-20 08:45:11,906 - INFO - Ingress ctest-nginx-ingress-75780102 verification passed 2026-02-20 08:45:11,909 - DEBUG - Pod ctest-busybox-pod-51550413 not present 2026-02-20 08:45:11,909 - INFO - Creating Pod ctest-busybox-pod-51550413 2026-02-20 08:45:11,932 - DEBUG - Node for Pod ctest-busybox-pod-51550413 not yet populated 2026-02-20 08:45:14,939 - DEBUG - Pod : ctest-busybox-pod-51550413 UUID is f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:14,950 - DEBUG - Pod ctest-nginx-pod-18170248 not in running state.Currently in Pending 2026-02-20 08:45:19,962 - INFO - Pod ctest-nginx-pod-18170248 is in running state.Got IP 10.47.255.250 2026-02-20 08:45:20,031 - DEBUG - Pod ctest-nginx-pod-18170248 has vmi 73ac44e4-0e38-11f1-8172-0242c0f44c93 2026-02-20 08:45:20,031 - INFO - Verified pod ctest-nginx-pod-18170248 in contrail-api 2026-02-20 08:45:20,031 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 08:45:20,043 - DEBUG - VMI 73ac44e4-0e38-11f1-8172-0242c0f44c93 is active in agent 10.0.0.41 2026-02-20 08:45:20,044 - INFO - Verified Pod ctest-nginx-pod-18170248 in agent 10.0.0.41 2026-02-20 08:45:20,044 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_VirtualMachineDatabaseList?x=ce441269-de00-4ac0-96e8-1c3382ef9626 2026-02-20 08:45:20,049 - INFO - Pod ctest-nginx-pod-18170248 with uuid ce441269-de00-4ac0-96e8-1c3382ef9626 found in kube manager 2026-02-20 08:45:20,049 - INFO - Pod ctest-nginx-pod-18170248 verification passed 2026-02-20 08:45:20,155 - DEBUG - [Pod ctest-nginx-pod-18170248] Cmd: echo ctest-nginx-pod-18170248 > /usr/share/nginx/html/index.html, Output: 2026-02-20 08:45:20,163 - INFO - Pod ctest-nginx-pod-87203779 is in running state.Got IP 10.47.255.249 2026-02-20 08:45:20,215 - DEBUG - Pod ctest-nginx-pod-87203779 has vmi 745ae1a2-0e38-11f1-8172-0242c0f44c93 2026-02-20 08:45:20,215 - INFO - Verified pod ctest-nginx-pod-87203779 in contrail-api 2026-02-20 08:45:20,215 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 08:45:20,225 - DEBUG - VMI 745ae1a2-0e38-11f1-8172-0242c0f44c93 is active in agent 10.0.0.41 2026-02-20 08:45:20,225 - INFO - Verified Pod ctest-nginx-pod-87203779 in agent 10.0.0.41 2026-02-20 08:45:20,225 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_VirtualMachineDatabaseList?x=d08b8776-e39d-4283-bc4d-c7cea36d8b69 2026-02-20 08:45:20,232 - INFO - Pod ctest-nginx-pod-87203779 with uuid d08b8776-e39d-4283-bc4d-c7cea36d8b69 found in kube manager 2026-02-20 08:45:20,232 - INFO - Pod ctest-nginx-pod-87203779 verification passed 2026-02-20 08:45:20,326 - DEBUG - [Pod ctest-nginx-pod-87203779] Cmd: echo ctest-nginx-pod-87203779 > /usr/share/nginx/html/index.html, Output: 2026-02-20 08:45:20,333 - DEBUG - Pod ctest-busybox-pod-51550413 not in running state.Currently in Pending 2026-02-20 08:45:25,341 - INFO - Pod ctest-busybox-pod-51550413 is in running state.Got IP 10.47.255.247 2026-02-20 08:45:25,394 - DEBUG - Pod ctest-busybox-pod-51550413 has vmi 797219d0-0e38-11f1-8172-0242c0f44c93 2026-02-20 08:45:25,394 - INFO - Verified pod ctest-busybox-pod-51550413 in contrail-api 2026-02-20 08:45:25,394 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-02-20 08:45:25,403 - DEBUG - VMI 797219d0-0e38-11f1-8172-0242c0f44c93 is active in agent 10.0.0.41 2026-02-20 08:45:25,403 - INFO - Verified Pod ctest-busybox-pod-51550413 in agent 10.0.0.41 2026-02-20 08:45:25,403 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_VirtualMachineDatabaseList?x=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:25,407 - INFO - Pod ctest-busybox-pod-51550413 with uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 found in kube manager 2026-02-20 08:45:25,407 - INFO - Pod ctest-busybox-pod-51550413 verification passed 2026-02-20 08:45:25,482 - DEBUG - [Pod ctest-busybox-pod-51550413] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-87203779 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 08:45:25,482 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-20 08:45:25,482 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd output: ctest-nginx-pod-87203779 Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 08:45:25,565 - DEBUG - [Pod ctest-busybox-pod-51550413] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-87203779 2026-02-20 08:45:25,565 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-20 08:45:25,565 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-87203779 2026-02-20 08:45:25,631 - DEBUG - [Pod ctest-busybox-pod-51550413] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-87203779 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 08:45:25,631 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-20 08:45:25,631 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-87203779 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 08:45:25,709 - DEBUG - [Pod ctest-busybox-pod-51550413] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-18170248 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 08:45:25,709 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-02-20 08:45:25,709 - DEBUG - [Pod Pod ctest-busybox-pod-51550413] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-18170248 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-02-20 08:45:25,709 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-18170248': 1, 'ctest-nginx-pod-87203779': 3} 2026-02-20 08:45:25,710 - WARNING - Nothing to delete parallely 2026-02-20 08:45:25,710 - INFO - Deleting pod default:ctest-busybox-pod-51550413 2026-02-20 08:45:25,720 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:25,727 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:45:30,727 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:30,733 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:45:35,734 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:35,739 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:45:40,740 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:40,747 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:45:45,748 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:45,754 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:45:50,754 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:50,761 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:45:55,761 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:45:55,767 - WARNING - Pod uuid f61839af-ec69-4222-8b2b-dbbbf288f9e1 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:46:00,768 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=f61839af-ec69-4222-8b2b-dbbbf288f9e1 2026-02-20 08:46:00,773 - DEBUG - Pod f61839af-ec69-4222-8b2b-dbbbf288f9e1 is not in agent 10.0.0.41 VM list 2026-02-20 08:46:00,773 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:797219d0-0e38-11f1-8172-0242c0f44c93 2026-02-20 08:46:00,778 - DEBUG - VMI 797219d0-0e38-11f1-8172-0242c0f44c93 is removed from agent 10.0.0.41 2026-02-20 08:46:00,778 - INFO - Verified that pod ctest-busybox-pod-51550413 is removed in agent 2026-02-20 08:46:00,778 - INFO - Deleting Ingress : ctest-nginx-ingress-75780102 2026-02-20 08:46:00,785 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_LoadbalancerDatabaseList?x=c186598a-3c83-43a5-ab23-c1b9704818fe 2026-02-20 08:46:00,790 - ERROR - Ingress ctest-nginx-ingress-75780102 with uuid c186598a-3c83-43a5-ab23-c1b9704818fe still found in kube manager 2026-02-20 08:46:01,791 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_LoadbalancerDatabaseList?x=c186598a-3c83-43a5-ab23-c1b9704818fe 2026-02-20 08:46:01,799 - ERROR - Ingress ctest-nginx-ingress-75780102 with uuid c186598a-3c83-43a5-ab23-c1b9704818fe still found in kube manager 2026-02-20 08:46:02,800 - DEBUG - Requesting: http://10.0.0.41:8108/Snh_LoadbalancerDatabaseList?x=c186598a-3c83-43a5-ab23-c1b9704818fe 2026-02-20 08:46:02,803 - INFO - Ingress ctest-nginx-ingress-75780102 with uuid c186598a-3c83-43a5-ab23-c1b9704818fe deleted successfully from kube manager 2026-02-20 08:46:02,803 - INFO - Verifications on Ingress %s cleanup passed 2026-02-20 08:46:02,804 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-02-20 08:46:02,976 - DEBUG - Requesting: http://10.0.0.41:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-02-20 08:46:02,981 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-02-20 08:46:02,981 - INFO - Deleting VN __public__ 2026-02-20 08:46:03,261 - DEBUG - Requesting: http://10.0.0.41:8082/routing-instance/63d1092f-269d-4eea-90dc-7ab114f94383 2026-02-20 08:46:03,273 - DEBUG - Response Code: 404 2026-02-20 08:46:03,273 - DEBUG - Requesting: http://10.0.0.41:8082/domains 2026-02-20 08:46:03,279 - DEBUG - Requesting: http://10.0.0.41:8082/domain/65444de8-4e2f-4931-97af-17702a2399e5 2026-02-20 08:46:03,285 - DEBUG - Requesting: http://10.0.0.41:8082/project/84460113-fc78-43e1-a90b-4ea2c8c6a334 2026-02-20 08:46:03,372 - INFO - Validated that VN __public__ is not found in API Server 2026-02-20 08:46:03,372 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-02-20 08:46:03,379 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:46:03,388 - DEBUG - VN __public__ is not present in Agent 10.0.0.41 2026-02-20 08:46:03,388 - INFO - Validated that VN __public__ is not in any agent 2026-02-20 08:46:03,388 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:46:03,393 - DEBUG - Requesting: http://10.0.0.41:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-02-20 08:46:03,406 - DEBUG - Requesting: http://10.0.0.41:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-02-20 08:46:03,411 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-02-20 08:46:03,411 - INFO - Deleting pod default:ctest-nginx-pod-87203779 2026-02-20 08:46:03,423 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=d08b8776-e39d-4283-bc4d-c7cea36d8b69 2026-02-20 08:46:03,441 - WARNING - Pod uuid d08b8776-e39d-4283-bc4d-c7cea36d8b69 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:46:08,441 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=d08b8776-e39d-4283-bc4d-c7cea36d8b69 2026-02-20 08:46:08,448 - DEBUG - Pod d08b8776-e39d-4283-bc4d-c7cea36d8b69 is not in agent 10.0.0.41 VM list 2026-02-20 08:46:08,448 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:745ae1a2-0e38-11f1-8172-0242c0f44c93 2026-02-20 08:46:08,457 - DEBUG - VMI 745ae1a2-0e38-11f1-8172-0242c0f44c93 is removed from agent 10.0.0.41 2026-02-20 08:46:08,458 - INFO - Verified that pod ctest-nginx-pod-87203779 is removed in agent 2026-02-20 08:46:08,458 - INFO - Deleting pod default:ctest-nginx-pod-18170248 2026-02-20 08:46:08,475 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ce441269-de00-4ac0-96e8-1c3382ef9626 2026-02-20 08:46:08,480 - WARNING - Pod uuid ce441269-de00-4ac0-96e8-1c3382ef9626 is still seen in agent 10.0.0.41 VM list 2026-02-20 08:46:13,481 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=ce441269-de00-4ac0-96e8-1c3382ef9626 2026-02-20 08:46:13,487 - DEBUG - Pod ce441269-de00-4ac0-96e8-1c3382ef9626 is not in agent 10.0.0.41 VM list 2026-02-20 08:46:13,487 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:73ac44e4-0e38-11f1-8172-0242c0f44c93 2026-02-20 08:46:13,492 - DEBUG - VMI 73ac44e4-0e38-11f1-8172-0242c0f44c93 is removed from agent 10.0.0.41 2026-02-20 08:46:13,492 - INFO - Verified that pod ctest-nginx-pod-18170248 is removed in agent 2026-02-20 08:46:13,493 - INFO - Deleting service : ctest-nginx-svc-44847957 2026-02-20 08:46:14,057 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2026-02-20 08:46:14,062 - DEBUG - No XMPP flaps were noticed during the test