2026-03-18 20:02:25,227 - DEBUG - [10.0.0.249]: Running cmd : hostname 2026-03-18 20:02:25,487 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-377-1 2026-03-18 20:02:25,488 - DEBUG - [10.0.0.249]: Running cmd : hostname -f 2026-03-18 20:02:25,608 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-377-1.cluster.local 2026-03-18 20:02:25,608 - DEBUG - [10.0.0.249]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2026-03-18 20:02:25,818 - DEBUG - Output : NAMES contrail_test_tvvXDXw2B k8s_autoscaler_dns-autoscaler-595558c478-f4b9f_kube-system_9c0d3ef9-ea7d-405c-ac89-4f222a47fa00_0 k8s_coredns_coredns-77f7cc69db-42l6x_kube-system_75363c2c-ffff-4d93-9dad-7e0c52aa0dba_0 k8s_opensdn-analytics-snmp-topology_opensdn-analytics-snmp-qcpf4_kube-system_d6b01aaf-c311-4174-aa16-47b332d272ae_1 k8s_opensdn-controller-config-nodemgr_opensdn-controller-config-7qvs7_kube-system_7809b613-a8cb-4378-8e9e-7bada4308fd8_0 k8s_opensdn-controller-config-svcmonitor_opensdn-controller-config-7qvs7_kube-system_7809b613-a8cb-4378-8e9e-7bada4308fd8_0 k8s_opensdn-controller-nodemgr_opensdn-controller-control-hb2kr_kube-system_c23d259a-22cd-4142-8983-fb1ca83b784c_0 k8s_opensdn-controller-control-named_opensdn-controller-control-hb2kr_kube-system_c23d259a-22cd-4142-8983-fb1ca83b784c_0 k8s_opensdn-analytics-query-engine_opensdn-analyticsdb-b95s8_kube-system_c1b76069-becc-4177-8b2c-f9942e99f05c_0 k8s_opensdn-controller-config-schema_opensdn-controller-config-7qvs7_kube-system_7809b613-a8cb-4378-8e9e-7bada4308fd8_0 k8s_opensdn-controller-control-dns_opensdn-controller-control-hb2kr_kube-system_c23d259a-22cd-4142-8983-fb1ca83b784c_0 k8s_opensdn-analyticsdb_opensdn-analyticsdb-b95s8_kube-system_c1b76069-becc-4177-8b2c-f9942e99f05c_0 k8s_opensdn-analytics-nodemgr_opensdn-analytics-s5zd9_kube-system_e4b8d45f-39d4-4d06-a370-aaddda250368_0 k8s_opensdn-analytics-collector_opensdn-analytics-s5zd9_kube-system_e4b8d45f-39d4-4d06-a370-aaddda250368_0 k8s_opensdn-controller-config-devicemgr_opensdn-controller-config-7qvs7_kube-system_7809b613-a8cb-4378-8e9e-7bada4308fd8_0 k8s_opensdn-analytics-snmp-nodemgr_opensdn-analytics-snmp-qcpf4_kube-system_d6b01aaf-c311-4174-aa16-47b332d272ae_0 k8s_opensdn-analytics-alarm-nodemgr_opensdn-analytics-alarm-2wf6h_kube-system_51e956bf-3fa3-4857-9c06-55726578c47f_0 k8s_opensdn-analytics-alarm-gen_opensdn-analytics-alarm-2wf6h_kube-system_51e956bf-3fa3-4857-9c06-55726578c47f_0 k8s_opensdn-controller-webui-web_opensdn-controller-webui-vkrxf_kube-system_a21ad6b5-aba5-48a5-9f49-701cc9ac08ac_0 k8s_opensdn-controller-control_opensdn-controller-control-hb2kr_kube-system_c23d259a-22cd-4142-8983-fb1ca83b784c_0 k8s_opensdn-analyticsdb-nodemgr_opensdn-analyticsdb-b95s8_kube-system_c1b76069-becc-4177-8b2c-f9942e99f05c_0 k8s_opensdn-analytics-api_opensdn-analytics-s5zd9_kube-system_e4b8d45f-39d4-4d06-a370-aaddda250368_0 k8s_opensdn-controller-config-api_opensdn-controller-config-7qvs7_kube-system_7809b613-a8cb-4378-8e9e-7bada4308fd8_0 k8s_opensdn-analytics-snmp-collector_opensdn-analytics-snmp-qcpf4_kube-system_d6b01aaf-c311-4174-aa16-47b332d272ae_0 k8s_kafka_opensdn-analytics-alarm-2wf6h_kube-system_51e956bf-3fa3-4857-9c06-55726578c47f_0 k8s_opensdn-controller-webui-job_opensdn-controller-webui-vkrxf_kube-system_a21ad6b5-aba5-48a5-9f49-701cc9ac08ac_0 k8s_redis_redis-w4phs_kube-system_64b6214a-0711-46e2-be29-e0590b9ab630_0 k8s_config-zookeeper_config-zookeeper-km2bc_kube-system_3d3839cc-b7e8-47cf-bce6-c07c325b3209_0 k8s_rabbitmq_rabbitmq-ldp98_kube-system_3310695c-d126-4ffa-abcc-a9e3b66c9e6e_0 k8s_opensdn-config-database-nodemgr_opensdn-configdb-vqfbd_kube-system_c7e9e0d2-daa3-417e-ad2a-ec0d302e3b7b_0 k8s_opensdn-configdb_opensdn-configdb-vqfbd_kube-system_c7e9e0d2-daa3-417e-ad2a-ec0d302e3b7b_0 k8s_opensdn-agent-nodemgr_opensdn-agent-gvvcd_kube-system_45dce4d1-1277-4c64-be93-39e14865771e_0 k8s_opensdn-vrouter-agent_opensdn-agent-gvvcd_kube-system_45dce4d1-1277-4c64-be93-39e14865771e_0 k8s_opensdn-kube-manager_opensdn-kube-manager-7gzj6_kube-system_6fcf46c2-19cd-4b1d-86c8-3b987e0431d6_0 k8s_kube-proxy_kube-proxy-l6cdx_kube-system_fb9fd86a-726c-4abc-9919-63c219a29cfe_0 k8s_kube-controller-manager_kube-controller-manager-cn-jenkins-deploy-platform-k8s-manifests-377-1_kube-system_e37c4d955a41355909e9aaf05470462c_1 k8s_kube-scheduler_kube-scheduler-cn-jenkins-deploy-platform-k8s-manifests-377-1_kube-system_cf2cf53e48c1ab6cf87df16cd6926912_1 k8s_kube-apiserver_kube-apiserver-cn-jenkins-deploy-platform-k8s-manifests-377-1_kube-system_80b818aafb6110bb42179e958c5f8253_0 2026-03-18 20:02:25,818 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2026-03-18 20:02:25,937 - DEBUG - Output : 127.0.0.1/8 172.17.0.1/16 10.96.0.3/32 10.96.0.1/32 10.0.0.249/24 2026-03-18 20:02:25,938 - DEBUG - [10.0.0.249]: Running cmd : getent hosts 10.0.0.249 | head -n 1 | awk '{print $2}' 2026-03-18 20:02:26,063 - DEBUG - Output : cn-jenkins-deploy-platform-k8s-manifests-377-1.cluster.local 2026-03-18 20:02:26,063 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2026-03-18 20:02:26,194 - DEBUG - Output : 10.0.0.249/24 2026-03-18 20:02:26,546 - INFO - Using existing project ['default-domain', 'k8s-default'](1679d9d3-4c50-4661-9072-ce327476718f) 2026-03-18 20:02:26,696 - INFO - Using existing project ['default-domain', 'k8s-default'](1679d9d3-4c50-4661-9072-ce327476718f) 2026-03-18 20:02:26,812 - INFO - ================================================================================ 2026-03-18 20:02:26,813 - INFO - STARTING TEST : test_ingress_ip_assignment 2026-03-18 20:02:26,813 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2026-03-18 20:02:27,094 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-18 20:02:27,099 - DEBUG - Nothing to compare xmpp stats {'10.0.0.249': {'10.0.0.249': '1'}} with 2026-03-18 20:02:27,099 - INFO - Initial checks done. Running the testcase now 2026-03-18 20:02:27,099 - INFO - 2026-03-18 20:02:27,103 - INFO - Namespace default already exists 2026-03-18 20:02:27,104 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-7gzj6_kube-system_6fcf46c2-19cd-4b1d-86c8-3b987e0431d6_0 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2026-03-18 20:02:27,267 - DEBUG - Output : cluster_project={} 2026-03-18 20:02:27,342 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2026-03-18 20:02:27,343 - DEBUG - Project uuid in Contrail is 1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:02:27,343 - DEBUG - [10.0.0.249]: Running cmd : docker exec --privileged -it k8s_opensdn-kube-manager_opensdn-kube-manager-7gzj6_kube-system_6fcf46c2-19cd-4b1d-86c8-3b987e0431d6_0 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2026-03-18 20:02:27,522 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2026-03-18 20:02:27,522 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_NamespaceDatabaseList?namespace_uuid=2b337c79-3ecc-4379-8ef6-af08bd3257f4 2026-03-18 20:02:27,527 - INFO - Namespace default with uuid 2b337c79-3ecc-4379-8ef6-af08bd3257f4 found in kube manager 2026-03-18 20:02:27,527 - INFO - Namespace default verification passed 2026-03-18 20:02:27,530 - DEBUG - Service ctest-nginx-svc-13371117 not present 2026-03-18 20:02:27,531 - INFO - Creating service ctest-nginx-svc-13371117 2026-03-18 20:02:27,543 - INFO - Created Service ctest-nginx-svc-13371117 2026-03-18 20:02:27,549 - DEBUG - Pod ctest-nginx-pod-21080274 not present 2026-03-18 20:02:27,549 - INFO - Creating Pod ctest-nginx-pod-21080274 2026-03-18 20:02:27,578 - DEBUG - Pod : ctest-nginx-pod-21080274 UUID is aa04f73d-50e7-4f2a-b11e-e41727ed7db2 2026-03-18 20:02:27,580 - DEBUG - Pod ctest-nginx-pod-63940452 not present 2026-03-18 20:02:27,581 - INFO - Creating Pod ctest-nginx-pod-63940452 2026-03-18 20:02:27,601 - DEBUG - Pod : ctest-nginx-pod-63940452 UUID is 531e7564-936b-49eb-88e8-64a290208810 2026-03-18 20:02:27,819 - INFO - Created VN __public__, UUID :d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:28,055 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-03-18 20:02:28,059 - DEBUG - Requesting: http://10.0.0.249:8082/domain/6509ba4e-da0c-4c75-bc3f-30887cae5f88 2026-03-18 20:02:28,070 - DEBUG - Requesting: http://10.0.0.249:8082/project/1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:02:28,106 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:28,119 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:28,127 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:28,133 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:28,139 - DEBUG - Route Targets: [] 2026-03-18 20:02:28,139 - DEBUG - RT names not yet present for VN __public__ 2026-03-18 20:02:33,140 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-03-18 20:02:33,145 - DEBUG - Requesting: http://10.0.0.249:8082/domain/6509ba4e-da0c-4c75-bc3f-30887cae5f88 2026-03-18 20:02:33,151 - DEBUG - Requesting: http://10.0.0.249:8082/project/1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:02:33,183 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,196 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,203 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:33,209 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:33,213 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/d4b36198-28e9-45fd-842e-2836271e62c5 2026-03-18 20:02:33,217 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-18 20:02:33,217 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,223 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:33,232 - INFO - Verified VN network id 11 for VN d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,232 - INFO - Verifications in API Server for VN __public__ passed 2026-03-18 20:02:33,232 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,239 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:33,244 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:02:33,248 - DEBUG - Requesting: http://10.0.0.249:8082/route-target/d4b36198-28e9-45fd-842e-2836271e62c5 2026-03-18 20:02:33,252 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-18 20:02:33,258 - DEBUG - Control-node 10.0.0.249 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15155193164574442574', 'uuid-lslong': '10534235287528595281'}, 'enable': 'true', 'created': '2026-03-18T20:02:27', 'last-modified': '2026-03-18T20:02:27', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.315101'} 2026-03-18 20:02:33,258 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:02:33,269 - DEBUG - Route Targets: ['target:64512:8000006'] 2026-03-18 20:02:33,269 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2026-03-18 20:02:33,270 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2026-03-18 20:02:33,270 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-03-18 20:02:33,274 - DEBUG - Requesting: http://10.0.0.249:8082/domain/6509ba4e-da0c-4c75-bc3f-30887cae5f88 2026-03-18 20:02:33,279 - DEBUG - Requesting: http://10.0.0.249:8082/project/1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:02:33,323 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,332 - DEBUG - =>VN __public__ has no policy to be verified 2026-03-18 20:02:33,332 - DEBUG - Verifying the vn in opserver 2026-03-18 20:02:33,332 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.249 2026-03-18 20:02:33,333 - DEBUG - Requesting: http://10.0.0.249:8081/analytics/uves/virtual-networks 2026-03-18 20:02:33,337 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.249:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2026-03-18 20:02:33,337 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2026-03-18 20:02:33,338 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:02:33,347 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:02:33,352 - DEBUG - Do not have enough data to verify VN in agent 2026-03-18 20:02:33,352 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:02:33,357 - DEBUG - VRF ids for VN __public__: {} 2026-03-18 20:02:33,537 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2026-03-18 20:02:33,750 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-03-18 20:02:33,754 - DEBUG - Requesting: http://10.0.0.249:8082/domain/6509ba4e-da0c-4c75-bc3f-30887cae5f88 2026-03-18 20:02:33,759 - DEBUG - Requesting: http://10.0.0.249:8082/project/1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:02:33,786 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,796 - DEBUG - Requesting: http://10.0.0.249:8082/floating-ip-pool/0b2af674-db6f-48e8-bf2a-0dbeef36f92c 2026-03-18 20:02:33,802 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-03-18 20:02:33,807 - DEBUG - Requesting: http://10.0.0.249:8082/domain/6509ba4e-da0c-4c75-bc3f-30887cae5f88 2026-03-18 20:02:33,813 - DEBUG - Requesting: http://10.0.0.249:8082/project/1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:02:33,863 - DEBUG - Requesting: http://10.0.0.249:8082/virtual-network/d2520fe3-34e3-444e-9231-1f2e6586df51 2026-03-18 20:02:33,881 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2026-03-18 20:02:33,881 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2026-03-18 20:02:33,904 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-18 20:02:33,911 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2026-03-18 20:02:33,911 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2026-03-18 20:02:33,916 - DEBUG - Ingress ctest-nginx-ingress-97211517 not present 2026-03-18 20:02:33,916 - INFO - Creating Ingress ctest-nginx-ingress-97211517 2026-03-18 20:02:33,936 - INFO - Created Ingress ctest-nginx-ingress-97211517 2026-03-18 20:02:33,954 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-97211517 2026-03-18 20:02:36,959 - DEBUG - For Ingress ctest-nginx-ingress-97211517, Cluster IP: 10.47.255.248, External IPs ['148.240.253.60'] 2026-03-18 20:02:36,960 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-97211517 2026-03-18 20:02:36,960 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=9dd32c19-4b91-44dd-b2fb-0f759a4d12ed 2026-03-18 20:02:36,963 - INFO - Ingress ctest-nginx-ingress-97211517 with uuid 9dd32c19-4b91-44dd-b2fb-0f759a4d12ed found in kube manager 2026-03-18 20:02:36,963 - INFO - Ingress ctest-nginx-ingress-97211517 verification passed 2026-03-18 20:02:36,966 - DEBUG - Pod ctest-busybox-pod-44514747 not present 2026-03-18 20:02:36,966 - INFO - Creating Pod ctest-busybox-pod-44514747 2026-03-18 20:02:36,983 - DEBUG - Pod : ctest-busybox-pod-44514747 UUID is c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:02:36,993 - DEBUG - Pod ctest-nginx-pod-21080274 not in running state.Currently in Pending 2026-03-18 20:02:42,002 - DEBUG - Pod ctest-nginx-pod-21080274 not in running state.Currently in Pending 2026-03-18 20:02:47,010 - INFO - Pod ctest-nginx-pod-21080274 is in running state.Got IP 10.47.255.250 2026-03-18 20:02:47,061 - DEBUG - Pod ctest-nginx-pod-21080274 has vmi 64f8069a-2305-11f1-ba20-024245e6e990 2026-03-18 20:02:47,061 - INFO - Verified pod ctest-nginx-pod-21080274 in contrail-api 2026-03-18 20:02:47,061 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-18 20:02:47,070 - DEBUG - VMI 64f8069a-2305-11f1-ba20-024245e6e990 is active in agent 10.0.0.249 2026-03-18 20:02:47,070 - INFO - Verified Pod ctest-nginx-pod-21080274 in agent 10.0.0.249 2026-03-18 20:02:47,070 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=aa04f73d-50e7-4f2a-b11e-e41727ed7db2 2026-03-18 20:02:47,074 - INFO - Pod ctest-nginx-pod-21080274 with uuid aa04f73d-50e7-4f2a-b11e-e41727ed7db2 found in kube manager 2026-03-18 20:02:47,074 - INFO - Pod ctest-nginx-pod-21080274 verification passed 2026-03-18 20:02:47,160 - DEBUG - [Pod ctest-nginx-pod-21080274] Cmd: echo ctest-nginx-pod-21080274 > /usr/share/nginx/html/index.html, Output: 2026-03-18 20:02:47,168 - INFO - Pod ctest-nginx-pod-63940452 is in running state.Got IP 10.47.255.249 2026-03-18 20:02:47,220 - DEBUG - Pod ctest-nginx-pod-63940452 has vmi 65a7b18a-2305-11f1-ba20-024245e6e990 2026-03-18 20:02:47,220 - INFO - Verified pod ctest-nginx-pod-63940452 in contrail-api 2026-03-18 20:02:47,220 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-18 20:02:47,231 - DEBUG - VMI 65a7b18a-2305-11f1-ba20-024245e6e990 is active in agent 10.0.0.249 2026-03-18 20:02:47,231 - INFO - Verified Pod ctest-nginx-pod-63940452 in agent 10.0.0.249 2026-03-18 20:02:47,231 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=531e7564-936b-49eb-88e8-64a290208810 2026-03-18 20:02:47,235 - INFO - Pod ctest-nginx-pod-63940452 with uuid 531e7564-936b-49eb-88e8-64a290208810 found in kube manager 2026-03-18 20:02:47,235 - INFO - Pod ctest-nginx-pod-63940452 verification passed 2026-03-18 20:02:47,390 - DEBUG - [Pod ctest-nginx-pod-63940452] Cmd: echo ctest-nginx-pod-63940452 > /usr/share/nginx/html/index.html, Output: 2026-03-18 20:02:47,398 - INFO - Pod ctest-busybox-pod-44514747 is in running state.Got IP 10.47.255.247 2026-03-18 20:02:47,452 - DEBUG - Pod ctest-busybox-pod-44514747 has vmi 69e9c634-2305-11f1-ba20-024245e6e990 2026-03-18 20:02:47,452 - INFO - Verified pod ctest-busybox-pod-44514747 in contrail-api 2026-03-18 20:02:47,452 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2026-03-18 20:02:47,459 - DEBUG - VMI 69e9c634-2305-11f1-ba20-024245e6e990 is active in agent 10.0.0.249 2026-03-18 20:02:47,459 - INFO - Verified Pod ctest-busybox-pod-44514747 in agent 10.0.0.249 2026-03-18 20:02:47,459 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_VirtualMachineDatabaseList?x=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:02:47,463 - INFO - Pod ctest-busybox-pod-44514747 with uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 found in kube manager 2026-03-18 20:02:47,463 - INFO - Pod ctest-busybox-pod-44514747 verification passed 2026-03-18 20:02:47,543 - DEBUG - [Pod ctest-busybox-pod-44514747] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-21080274 2026-03-18 20:02:47,543 - DEBUG - [Pod Pod ctest-busybox-pod-44514747] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-18 20:02:47,543 - DEBUG - [Pod Pod ctest-busybox-pod-44514747] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-21080274 2026-03-18 20:02:47,622 - DEBUG - [Pod ctest-busybox-pod-44514747] Cmd: wget http://10.47.255.248:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-63940452 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-18 20:02:47,622 - DEBUG - [Pod Pod ctest-busybox-pod-44514747] Cmd wget http://10.47.255.248:80/ -O - -T 5 -t 1 passed 2026-03-18 20:02:47,622 - DEBUG - [Pod Pod ctest-busybox-pod-44514747] Cmd output: Connecting to 10.47.255.248:80 (10.47.255.248:80) ctest-nginx-pod-63940452 writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2026-03-18 20:02:47,622 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-21080274': 1, 'ctest-nginx-pod-63940452': 1} 2026-03-18 20:02:47,622 - WARNING - Nothing to delete parallely 2026-03-18 20:02:47,623 - INFO - Deleting pod default:ctest-busybox-pod-44514747 2026-03-18 20:02:47,638 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:02:47,646 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:02:52,647 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:02:52,652 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:02:57,653 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:02:57,659 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:02,659 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:03:02,664 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:07,665 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:03:07,670 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:12,671 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:03:12,676 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:17,677 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:03:17,683 - WARNING - Pod uuid c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:22,684 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c2ba216c-6aa4-47d2-8d0a-0572ef08be10 2026-03-18 20:03:22,689 - DEBUG - Pod c2ba216c-6aa4-47d2-8d0a-0572ef08be10 is not in agent 10.0.0.249 VM list 2026-03-18 20:03:22,689 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:69e9c634-2305-11f1-ba20-024245e6e990 2026-03-18 20:03:22,693 - DEBUG - VMI 69e9c634-2305-11f1-ba20-024245e6e990 is removed from agent 10.0.0.249 2026-03-18 20:03:22,694 - INFO - Verified that pod ctest-busybox-pod-44514747 is removed in agent 2026-03-18 20:03:22,694 - INFO - Deleting Ingress : ctest-nginx-ingress-97211517 2026-03-18 20:03:22,699 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=9dd32c19-4b91-44dd-b2fb-0f759a4d12ed 2026-03-18 20:03:22,703 - ERROR - Ingress ctest-nginx-ingress-97211517 with uuid 9dd32c19-4b91-44dd-b2fb-0f759a4d12ed still found in kube manager 2026-03-18 20:03:23,703 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=9dd32c19-4b91-44dd-b2fb-0f759a4d12ed 2026-03-18 20:03:23,707 - ERROR - Ingress ctest-nginx-ingress-97211517 with uuid 9dd32c19-4b91-44dd-b2fb-0f759a4d12ed still found in kube manager 2026-03-18 20:03:24,708 - DEBUG - Requesting: http://10.0.0.249:8108/Snh_LoadbalancerDatabaseList?x=9dd32c19-4b91-44dd-b2fb-0f759a4d12ed 2026-03-18 20:03:24,714 - INFO - Ingress ctest-nginx-ingress-97211517 with uuid 9dd32c19-4b91-44dd-b2fb-0f759a4d12ed deleted successfully from kube manager 2026-03-18 20:03:24,714 - INFO - Verifications on Ingress %s cleanup passed 2026-03-18 20:03:24,714 - INFO - Deleting the FIP pool __fip_pool_public__ 2026-03-18 20:03:24,800 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2026-03-18 20:03:24,804 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2026-03-18 20:03:24,805 - INFO - Deleting VN __public__ 2026-03-18 20:03:24,881 - DEBUG - Requesting: http://10.0.0.249:8082/routing-instance/b76f7dd9-6f7f-4ec1-93b3-de314da8ad72 2026-03-18 20:03:24,890 - DEBUG - Response Code: 404 2026-03-18 20:03:24,890 - DEBUG - Requesting: http://10.0.0.249:8082/domains 2026-03-18 20:03:24,896 - DEBUG - Requesting: http://10.0.0.249:8082/domain/6509ba4e-da0c-4c75-bc3f-30887cae5f88 2026-03-18 20:03:24,905 - DEBUG - Requesting: http://10.0.0.249:8082/project/1679d9d3-4c50-4661-9072-ce327476718f 2026-03-18 20:03:24,945 - INFO - Validated that VN __public__ is not found in API Server 2026-03-18 20:03:24,945 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2026-03-18 20:03:24,950 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:03:24,955 - DEBUG - VN __public__ is not present in Agent 10.0.0.249 2026-03-18 20:03:24,955 - INFO - Validated that VN __public__ is not in any agent 2026-03-18 20:03:24,955 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:03:24,960 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2026-03-18 20:03:24,968 - DEBUG - Requesting: http://10.0.0.249:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2026-03-18 20:03:24,972 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2026-03-18 20:03:24,973 - INFO - Deleting pod default:ctest-nginx-pod-63940452 2026-03-18 20:03:24,984 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=531e7564-936b-49eb-88e8-64a290208810 2026-03-18 20:03:24,991 - WARNING - Pod uuid 531e7564-936b-49eb-88e8-64a290208810 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:29,992 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=531e7564-936b-49eb-88e8-64a290208810 2026-03-18 20:03:29,996 - DEBUG - Pod 531e7564-936b-49eb-88e8-64a290208810 is not in agent 10.0.0.249 VM list 2026-03-18 20:03:29,996 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:65a7b18a-2305-11f1-ba20-024245e6e990 2026-03-18 20:03:30,002 - DEBUG - VMI 65a7b18a-2305-11f1-ba20-024245e6e990 is removed from agent 10.0.0.249 2026-03-18 20:03:30,002 - INFO - Verified that pod ctest-nginx-pod-63940452 is removed in agent 2026-03-18 20:03:30,002 - INFO - Deleting pod default:ctest-nginx-pod-21080274 2026-03-18 20:03:30,012 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=aa04f73d-50e7-4f2a-b11e-e41727ed7db2 2026-03-18 20:03:30,019 - WARNING - Pod uuid aa04f73d-50e7-4f2a-b11e-e41727ed7db2 is still seen in agent 10.0.0.249 VM list 2026-03-18 20:03:35,019 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=aa04f73d-50e7-4f2a-b11e-e41727ed7db2 2026-03-18 20:03:35,024 - DEBUG - Pod aa04f73d-50e7-4f2a-b11e-e41727ed7db2 is not in agent 10.0.0.249 VM list 2026-03-18 20:03:35,024 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:64f8069a-2305-11f1-ba20-024245e6e990 2026-03-18 20:03:35,027 - DEBUG - VMI 64f8069a-2305-11f1-ba20-024245e6e990 is removed from agent 10.0.0.249 2026-03-18 20:03:35,028 - INFO - Verified that pod ctest-nginx-pod-21080274 is removed in agent 2026-03-18 20:03:35,028 - INFO - Deleting service : ctest-nginx-svc-13371117 2026-03-18 20:03:35,572 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq? 2026-03-18 20:03:35,577 - DEBUG - No XMPP flaps were noticed during the test