2025-10-24 16:57:30,048 - DEBUG - [10.0.0.19]: Running cmd : hostname 2025-10-24 16:57:30,286 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1612-1 2025-10-24 16:57:30,286 - DEBUG - [10.0.0.19]: Running cmd : hostname -f 2025-10-24 16:57:30,320 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1612-1 2025-10-24 16:57:30,320 - DEBUG - [10.0.0.19]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-24 16:57:30,414 - DEBUG - Output : NAMES contrail_test_mqfgRAp62 contrail-kubernetes-master-kubemanager-1 analytics_database-cassandra-1 analytics_database-query-engine-1 analytics_database-nodemgr-1 analytics_snmp-topology-1 analytics_snmp-snmp-collector-1 analytics_snmp-nodemgr-1 analytics_alarm-nodemgr-1 analytics_alarm-alarm-gen-1 analytics_alarm-kafka-1 analytics-nodemgr-1 analytics-collector-1 analytics-api-1 redis-redis-1 webui-job-1 webui-web-1 control-named-1 control-dns-1 control-nodemgr-1 control-control-1 config_database-zookeeper-1 config_database-rabbitmq-1 config_database-cassandra-1 config_database-nodemgr-1 config_api-devicemgr-1 config_api-api-1 config_api-svcmonitor-1 config_api-schema-1 config_api-dnsmasq-1 config_api-nodemgr-1 2025-10-24 16:57:30,414 - DEBUG - [10.0.0.19]: Running cmd : ip -4 -o addr show | awk '{print $4}' 2025-10-24 16:57:30,447 - DEBUG - Output : 127.0.0.1/8 10.0.0.19/24 252.19.0.1/8 10.233.19.1/24 172.17.0.1/16 2025-10-24 16:57:30,447 - DEBUG - [10.0.0.19]: Running cmd : getent hosts 10.0.0.19 | head -n 1 | awk '{print $2}' 2025-10-24 16:57:30,527 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1612-1 2025-10-24 16:57:30,584 - DEBUG - [10.0.0.41]: Running cmd : hostname 2025-10-24 16:57:30,826 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1612-1 2025-10-24 16:57:30,826 - DEBUG - [10.0.0.41]: Running cmd : hostname -f 2025-10-24 16:57:30,861 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1612-1 2025-10-24 16:57:30,861 - DEBUG - [10.0.0.41]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-10-24 16:57:30,906 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2025-10-24 16:57:30,906 - DEBUG - [10.0.0.41]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}' 2025-10-24 16:57:30,981 - DEBUG - Output : 10.0.0.41/24 2025-10-24 16:57:30,981 - DEBUG - [10.0.0.41]: Running cmd : getent hosts 10.0.0.41 | head -n 1 | awk '{print $2}' 2025-10-24 16:57:31,015 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1612-1 2025-10-24 16:57:31,079 - INFO - Using existing project ['default-domain', 'k8s-default'](8338bdf7-8c86-48cb-9d67-b51ae5a37b11) 2025-10-24 16:57:31,180 - INFO - Using existing project ['default-domain', 'k8s-default'](8338bdf7-8c86-48cb-9d67-b51ae5a37b11) 2025-10-24 16:57:31,308 - INFO - ================================================================================ 2025-10-24 16:57:31,308 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-10-24 16:57:31,308 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-10-24 16:57:31,499 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-24 16:57:31,507 - DEBUG - Nothing to compare xmpp stats {'10.0.0.41': {'10.0.0.19': '0'}} with 2025-10-24 16:57:31,507 - INFO - Initial checks done. Running the testcase now 2025-10-24 16:57:31,508 - INFO - 2025-10-24 16:57:31,512 - INFO - Namespace default already exists 2025-10-24 16:57:31,513 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-10-24 16:57:31,629 - DEBUG - Output : cluster_project={} 2025-10-24 16:57:31,695 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-10-24 16:57:31,695 - DEBUG - Project uuid in Contrail is 8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:57:31,696 - DEBUG - [10.0.0.19]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-10-24 16:57:31,846 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-10-24 16:57:31,846 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_NamespaceDatabaseList?namespace_uuid=0d8b6553-2172-4c46-b8d0-96931d070e5f 2025-10-24 16:57:31,850 - INFO - Namespace default with uuid 0d8b6553-2172-4c46-b8d0-96931d070e5f found in kube manager 2025-10-24 16:57:31,850 - INFO - Namespace default verification passed 2025-10-24 16:57:31,853 - DEBUG - Service ctest-nginx-svc-75301741 not present 2025-10-24 16:57:31,854 - INFO - Creating service ctest-nginx-svc-75301741 2025-10-24 16:57:31,875 - INFO - Created Service ctest-nginx-svc-75301741 2025-10-24 16:57:31,881 - DEBUG - Pod ctest-nginx-pod-72299221 not present 2025-10-24 16:57:31,881 - INFO - Creating Pod ctest-nginx-pod-72299221 2025-10-24 16:57:31,950 - DEBUG - Pod : ctest-nginx-pod-72299221 UUID is a90f0f95-fb7c-4257-8fea-0841f7aff8be 2025-10-24 16:57:31,964 - DEBUG - Pod ctest-nginx-pod-71724605 not present 2025-10-24 16:57:31,964 - INFO - Creating Pod ctest-nginx-pod-71724605 2025-10-24 16:57:31,998 - DEBUG - Pod : ctest-nginx-pod-71724605 UUID is 3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 2025-10-24 16:57:32,141 - INFO - Created VN __public__, UUID :e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:32,400 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-24 16:57:32,405 - DEBUG - Requesting: http://10.0.0.19:8082/domain/2354dfd1-3515-4f14-85eb-4072c36af627 2025-10-24 16:57:32,417 - DEBUG - Requesting: http://10.0.0.19:8082/project/8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:57:32,463 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:32,475 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:32,485 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:32,492 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:32,498 - DEBUG - Route Targets: [] 2025-10-24 16:57:32,498 - DEBUG - RT names not yet present for VN __public__ 2025-10-24 16:57:37,498 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-24 16:57:37,504 - DEBUG - Requesting: http://10.0.0.19:8082/domain/2354dfd1-3515-4f14-85eb-4072c36af627 2025-10-24 16:57:37,510 - DEBUG - Requesting: http://10.0.0.19:8082/project/8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:57:37,553 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:37,563 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:37,573 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:37,582 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:37,589 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/39abac09-e2c5-41df-8c48-b6c185391a6c 2025-10-24 16:57:37,595 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-24 16:57:37,596 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:37,604 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:37,617 - INFO - Verified VN network id 13 for VN e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:37,617 - INFO - Verifications in API Server for VN __public__ passed 2025-10-24 16:57:37,617 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:37,625 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:37,631 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:57:37,637 - DEBUG - Requesting: http://10.0.0.19:8082/route-target/39abac09-e2c5-41df-8c48-b6c185391a6c 2025-10-24 16:57:37,646 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-24 16:57:37,655 - DEBUG - Control-node 10.0.0.19 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '16642350187259972258', 'uuid-lslong': '9478887684138431442'}, 'enable': 'true', 'created': '2025-10-24T16:57:32', 'last-modified': '2025-10-24T16:57:32', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:05.349109'} 2025-10-24 16:57:37,655 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:57:37,666 - DEBUG - Route Targets: ['target:64512:8000006'] 2025-10-24 16:57:37,666 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-10-24 16:57:37,667 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-10-24 16:57:37,667 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-24 16:57:37,673 - DEBUG - Requesting: http://10.0.0.19:8082/domain/2354dfd1-3515-4f14-85eb-4072c36af627 2025-10-24 16:57:37,681 - DEBUG - Requesting: http://10.0.0.19:8082/project/8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:57:37,734 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:37,743 - DEBUG - =>VN __public__ has no policy to be verified 2025-10-24 16:57:37,743 - DEBUG - Verifying the vn in opserver 2025-10-24 16:57:37,743 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.19 2025-10-24 16:57:37,743 - DEBUG - Requesting: http://10.0.0.19:8081/analytics/uves/virtual-networks 2025-10-24 16:57:37,747 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.19:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-10-24 16:57:37,747 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-10-24 16:57:37,747 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:57:37,756 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:57:37,761 - DEBUG - Do not have enough data to verify VN in agent 2025-10-24 16:57:37,761 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:57:37,766 - DEBUG - VRF ids for VN __public__: {} 2025-10-24 16:57:37,850 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-10-24 16:57:38,000 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-24 16:57:38,005 - DEBUG - Requesting: http://10.0.0.19:8082/domain/2354dfd1-3515-4f14-85eb-4072c36af627 2025-10-24 16:57:38,012 - DEBUG - Requesting: http://10.0.0.19:8082/project/8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:57:38,051 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:38,061 - DEBUG - Requesting: http://10.0.0.19:8082/floating-ip-pool/1e5eddae-703f-4034-b0e5-ae788e28fc16 2025-10-24 16:57:38,067 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-24 16:57:38,072 - DEBUG - Requesting: http://10.0.0.19:8082/domain/2354dfd1-3515-4f14-85eb-4072c36af627 2025-10-24 16:57:38,079 - DEBUG - Requesting: http://10.0.0.19:8082/project/8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:57:38,127 - DEBUG - Requesting: http://10.0.0.19:8082/virtual-network/e6f58152-0349-4aa2-838b-c6209ee573d2 2025-10-24 16:57:38,135 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-10-24 16:57:38,135 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-10-24 16:57:38,142 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-24 16:57:38,148 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-10-24 16:57:38,148 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-10-24 16:57:38,150 - DEBUG - Ingress ctest-nginx-ingress-30705993 not present 2025-10-24 16:57:38,151 - INFO - Creating Ingress ctest-nginx-ingress-30705993 2025-10-24 16:57:38,158 - INFO - Created Ingress ctest-nginx-ingress-30705993 2025-10-24 16:57:38,161 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-30705993 2025-10-24 16:57:41,166 - DEBUG - For Ingress ctest-nginx-ingress-30705993, Cluster IP: 10.47.255.249, External IPs ['98.73.228.188'] 2025-10-24 16:57:41,166 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-30705993 2025-10-24 16:57:41,166 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=d02a2625-66c8-43ad-b30c-b1f240623c32 2025-10-24 16:57:41,170 - INFO - Ingress ctest-nginx-ingress-30705993 with uuid d02a2625-66c8-43ad-b30c-b1f240623c32 found in kube manager 2025-10-24 16:57:41,171 - INFO - Ingress ctest-nginx-ingress-30705993 verification passed 2025-10-24 16:57:41,173 - DEBUG - Pod ctest-busybox-pod-21664209 not present 2025-10-24 16:57:41,174 - INFO - Creating Pod ctest-busybox-pod-21664209 2025-10-24 16:57:41,201 - DEBUG - Pod : ctest-busybox-pod-21664209 UUID is e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:57:41,228 - DEBUG - Pod ctest-nginx-pod-72299221 not in running state.Currently in Pending 2025-10-24 16:57:46,237 - INFO - Pod ctest-nginx-pod-72299221 is in running state.Got IP 10.47.255.251 2025-10-24 16:57:46,276 - DEBUG - Pod ctest-nginx-pod-72299221 has vmi 88e701a0-b0fa-11f0-87e3-fa163e9d66fe 2025-10-24 16:57:46,276 - INFO - Verified pod ctest-nginx-pod-72299221 in contrail-api 2025-10-24 16:57:46,276 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-24 16:57:46,288 - DEBUG - VMI 88e701a0-b0fa-11f0-87e3-fa163e9d66fe is active in agent 10.0.0.41 2025-10-24 16:57:46,288 - INFO - Verified Pod ctest-nginx-pod-72299221 in agent 10.0.0.41 2025-10-24 16:57:46,289 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=a90f0f95-fb7c-4257-8fea-0841f7aff8be 2025-10-24 16:57:46,293 - INFO - Pod ctest-nginx-pod-72299221 with uuid a90f0f95-fb7c-4257-8fea-0841f7aff8be found in kube manager 2025-10-24 16:57:46,293 - INFO - Pod ctest-nginx-pod-72299221 verification passed 2025-10-24 16:57:46,376 - DEBUG - [Pod ctest-nginx-pod-72299221] Cmd: echo ctest-nginx-pod-72299221 > /usr/share/nginx/html/index.html, Output: 2025-10-24 16:57:46,382 - INFO - Pod ctest-nginx-pod-71724605 is in running state.Got IP 10.47.255.250 2025-10-24 16:57:46,429 - DEBUG - Pod ctest-nginx-pod-71724605 has vmi 89510618-b0fa-11f0-87e3-fa163e9d66fe 2025-10-24 16:57:46,429 - INFO - Verified pod ctest-nginx-pod-71724605 in contrail-api 2025-10-24 16:57:46,430 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-24 16:57:46,457 - DEBUG - VMI 89510618-b0fa-11f0-87e3-fa163e9d66fe is active in agent 10.0.0.41 2025-10-24 16:57:46,457 - INFO - Verified Pod ctest-nginx-pod-71724605 in agent 10.0.0.41 2025-10-24 16:57:46,458 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 2025-10-24 16:57:46,466 - INFO - Pod ctest-nginx-pod-71724605 with uuid 3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 found in kube manager 2025-10-24 16:57:46,467 - INFO - Pod ctest-nginx-pod-71724605 verification passed 2025-10-24 16:57:46,532 - DEBUG - [Pod ctest-nginx-pod-71724605] Cmd: echo ctest-nginx-pod-71724605 > /usr/share/nginx/html/index.html, Output: 2025-10-24 16:57:46,546 - DEBUG - Pod ctest-busybox-pod-21664209 not in running state.Currently in Pending 2025-10-24 16:57:51,554 - DEBUG - Pod ctest-busybox-pod-21664209 not in running state.Currently in Pending 2025-10-24 16:57:56,561 - INFO - Pod ctest-busybox-pod-21664209 is in running state.Got IP 10.47.255.248 2025-10-24 16:57:56,616 - DEBUG - Pod ctest-busybox-pod-21664209 has vmi 8dafed96-b0fa-11f0-87e3-fa163e9d66fe 2025-10-24 16:57:56,616 - INFO - Verified pod ctest-busybox-pod-21664209 in contrail-api 2025-10-24 16:57:56,616 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-10-24 16:57:56,626 - DEBUG - VMI 8dafed96-b0fa-11f0-87e3-fa163e9d66fe is active in agent 10.0.0.41 2025-10-24 16:57:56,626 - INFO - Verified Pod ctest-busybox-pod-21664209 in agent 10.0.0.41 2025-10-24 16:57:56,626 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_VirtualMachineDatabaseList?x=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:57:56,630 - INFO - Pod ctest-busybox-pod-21664209 with uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 found in kube manager 2025-10-24 16:57:56,630 - INFO - Pod ctest-busybox-pod-21664209 verification passed 2025-10-24 16:57:56,693 - DEBUG - [Pod ctest-busybox-pod-21664209] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-72299221 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-24 16:57:56,693 - DEBUG - [Pod Pod ctest-busybox-pod-21664209] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-24 16:57:56,693 - DEBUG - [Pod Pod ctest-busybox-pod-21664209] Cmd output: ctest-nginx-pod-72299221 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-24 16:57:56,751 - DEBUG - [Pod ctest-busybox-pod-21664209] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-71724605 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-24 16:57:56,751 - DEBUG - [Pod Pod ctest-busybox-pod-21664209] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-10-24 16:57:56,751 - DEBUG - [Pod Pod ctest-busybox-pod-21664209] Cmd output: ctest-nginx-pod-71724605 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-10-24 16:57:56,751 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-72299221': 1, 'ctest-nginx-pod-71724605': 1} 2025-10-24 16:57:56,752 - WARNING - Nothing to delete parallely 2025-10-24 16:57:56,752 - INFO - Deleting pod default:ctest-busybox-pod-21664209 2025-10-24 16:57:56,765 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:57:56,772 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:01,772 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:01,779 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:06,780 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:06,787 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:11,787 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:11,793 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:16,795 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:16,806 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:21,807 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:21,816 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:26,816 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:26,824 - WARNING - Pod uuid e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:31,825 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 2025-10-24 16:58:31,831 - DEBUG - Pod e85f2e3a-435c-44f4-bc9c-03aec4e4a2e1 is not in agent 10.0.0.41 VM list 2025-10-24 16:58:31,831 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:8dafed96-b0fa-11f0-87e3-fa163e9d66fe 2025-10-24 16:58:31,836 - DEBUG - VMI 8dafed96-b0fa-11f0-87e3-fa163e9d66fe is removed from agent 10.0.0.41 2025-10-24 16:58:31,836 - INFO - Verified that pod ctest-busybox-pod-21664209 is removed in agent 2025-10-24 16:58:31,836 - INFO - Deleting Ingress : ctest-nginx-ingress-30705993 2025-10-24 16:58:31,845 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=d02a2625-66c8-43ad-b30c-b1f240623c32 2025-10-24 16:58:31,850 - ERROR - Ingress ctest-nginx-ingress-30705993 with uuid d02a2625-66c8-43ad-b30c-b1f240623c32 still found in kube manager 2025-10-24 16:58:32,853 - DEBUG - Requesting: http://10.0.0.19:8108/Snh_LoadbalancerDatabaseList?x=d02a2625-66c8-43ad-b30c-b1f240623c32 2025-10-24 16:58:32,859 - INFO - Ingress ctest-nginx-ingress-30705993 with uuid d02a2625-66c8-43ad-b30c-b1f240623c32 deleted successfully from kube manager 2025-10-24 16:58:32,860 - INFO - Verifications on Ingress %s cleanup passed 2025-10-24 16:58:32,861 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-10-24 16:58:32,964 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-10-24 16:58:32,974 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-10-24 16:58:32,975 - INFO - Deleting VN __public__ 2025-10-24 16:58:33,064 - DEBUG - Requesting: http://10.0.0.19:8082/routing-instance/6b542dae-8a75-4f52-bd06-d1420f1e47db 2025-10-24 16:58:33,068 - DEBUG - Response Code: 404 2025-10-24 16:58:33,068 - DEBUG - Requesting: http://10.0.0.19:8082/domains 2025-10-24 16:58:33,074 - DEBUG - Requesting: http://10.0.0.19:8082/domain/2354dfd1-3515-4f14-85eb-4072c36af627 2025-10-24 16:58:33,083 - DEBUG - Requesting: http://10.0.0.19:8082/project/8338bdf7-8c86-48cb-9d67-b51ae5a37b11 2025-10-24 16:58:33,146 - INFO - Validated that VN __public__ is not found in API Server 2025-10-24 16:58:33,146 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-10-24 16:58:33,152 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:58:33,168 - DEBUG - VN __public__ is not present in Agent 10.0.0.41 2025-10-24 16:58:33,168 - INFO - Validated that VN __public__ is not in any agent 2025-10-24 16:58:33,168 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:58:33,189 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-10-24 16:58:33,194 - DEBUG - Requesting: http://10.0.0.19:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-10-24 16:58:33,200 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-10-24 16:58:33,200 - INFO - Deleting pod default:ctest-nginx-pod-71724605 2025-10-24 16:58:33,227 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 2025-10-24 16:58:33,240 - WARNING - Pod uuid 3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:38,241 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 2025-10-24 16:58:38,247 - DEBUG - Pod 3e6d76e3-9de8-4ebe-bf3d-d79f4d360c11 is not in agent 10.0.0.41 VM list 2025-10-24 16:58:38,247 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:89510618-b0fa-11f0-87e3-fa163e9d66fe 2025-10-24 16:58:38,252 - DEBUG - VMI 89510618-b0fa-11f0-87e3-fa163e9d66fe is removed from agent 10.0.0.41 2025-10-24 16:58:38,252 - INFO - Verified that pod ctest-nginx-pod-71724605 is removed in agent 2025-10-24 16:58:38,252 - INFO - Deleting pod default:ctest-nginx-pod-72299221 2025-10-24 16:58:38,264 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=a90f0f95-fb7c-4257-8fea-0841f7aff8be 2025-10-24 16:58:38,273 - WARNING - Pod uuid a90f0f95-fb7c-4257-8fea-0841f7aff8be is still seen in agent 10.0.0.41 VM list 2025-10-24 16:58:43,274 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_VmListReq?uuid=a90f0f95-fb7c-4257-8fea-0841f7aff8be 2025-10-24 16:58:43,280 - DEBUG - Pod a90f0f95-fb7c-4257-8fea-0841f7aff8be is not in agent 10.0.0.41 VM list 2025-10-24 16:58:43,280 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:88e701a0-b0fa-11f0-87e3-fa163e9d66fe 2025-10-24 16:58:43,287 - DEBUG - VMI 88e701a0-b0fa-11f0-87e3-fa163e9d66fe is removed from agent 10.0.0.41 2025-10-24 16:58:43,287 - INFO - Verified that pod ctest-nginx-pod-72299221 is removed in agent 2025-10-24 16:58:43,288 - INFO - Deleting service : ctest-nginx-svc-75301741 2025-10-24 16:58:43,503 - DEBUG - Requesting: http://10.0.0.41:8085/Snh_AgentXmppConnectionStatusReq? 2025-10-24 16:58:43,509 - DEBUG - No XMPP flaps were noticed during the test