2025-06-04 14:15:52,192 - DEBUG - [10.0.0.38]: Running cmd : hostname 2025-06-04 14:15:52,445 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1249-1 2025-06-04 14:15:52,445 - DEBUG - [10.0.0.38]: Running cmd : hostname -f 2025-06-04 14:15:52,478 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1249-1 2025-06-04 14:15:52,478 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-06-04 14:15:52,536 - DEBUG - Output : NAMES contrail_test_JmqJNALEv contrail-kubernetes-master-kubemanager-1 analytics_database-nodemgr-1 analytics_database-query-engine-1 analytics_database-cassandra-1 analytics_snmp-topology-1 analytics_snmp-nodemgr-1 analytics_snmp-snmp-collector-1 analytics_alarm-kafka-1 analytics_alarm-alarm-gen-1 analytics_alarm-nodemgr-1 analytics-collector-1 analytics-api-1 analytics-nodemgr-1 redis-redis-1 webui-web-1 webui-job-1 control-named-1 control-nodemgr-1 control-dns-1 control-control-1 config_database-nodemgr-1 config_database-rabbitmq-1 config_database-zookeeper-1 config_database-cassandra-1 config_api-devicemgr-1 config_api-svcmonitor-1 config_api-schema-1 config_api-nodemgr-1 config_api-dnsmasq-1 config_api-api-1 2025-06-04 14:15:52,537 - DEBUG - [10.0.0.38]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-06-04 14:15:52,610 - DEBUG - Output : 10.0.0.38/24 172.17.0.1/16 2025-06-04 14:15:52,610 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}' 2025-06-04 14:15:52,644 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1249-1 2025-06-04 14:15:52,734 - DEBUG - [10.0.0.57]: Running cmd : hostname 2025-06-04 14:15:53,032 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1249-1 2025-06-04 14:15:53,032 - DEBUG - [10.0.0.57]: Running cmd : hostname -f 2025-06-04 14:15:53,073 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1249-1 2025-06-04 14:15:53,073 - DEBUG - [10.0.0.57]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}' 2025-06-04 14:15:53,148 - DEBUG - Output : NAMES vrouter-vrouter-agent-1 vrouter-nodemgr-1 2025-06-04 14:15:53,149 - DEBUG - [10.0.0.57]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}' 2025-06-04 14:15:53,182 - DEBUG - Output : 10.0.0.57/24 2025-06-04 14:15:53,182 - DEBUG - [10.0.0.57]: Running cmd : getent hosts 10.0.0.57 | head -n 1 | awk '{print $2}' 2025-06-04 14:15:53,217 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1249-1 2025-06-04 14:15:53,350 - INFO - Using existing project ['default-domain', 'k8s-default'](7dc6db24-cf2b-4152-86d7-419fe1fbfa16) 2025-06-04 14:15:53,464 - INFO - Using existing project ['default-domain', 'k8s-default'](7dc6db24-cf2b-4152-86d7-419fe1fbfa16) 2025-06-04 14:15:53,622 - INFO - ================================================================================ 2025-06-04 14:15:53,622 - INFO - STARTING TEST : test_ingress_ip_assignment 2025-06-04 14:15:53,623 - INFO - TEST DESCRIPTION : Verify that Ingress gets a CLuster IP which is reachable to Pods in same namespace. Also verify that a Floating IP is assigned to the Ingress from the Public FIP poo. Steps: 1. Create a service with 2 pods running nginx 2. Create an ingress out of this service 3. From another Pod do a wget on the ingress Cluster ip Validate that Ingress get a IP from Public FIP pool which might/might not be accessible. Validate that service and its loadbalancing work 2025-06-04 14:15:53,825 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AgentXmppConnectionStatusReq? 2025-06-04 14:15:53,834 - DEBUG - Nothing to compare xmpp stats {'10.0.0.57': {'10.0.0.38': '0'}} with 2025-06-04 14:15:53,834 - INFO - Initial checks done. Running the testcase now 2025-06-04 14:15:53,835 - INFO - 2025-06-04 14:15:53,840 - INFO - Namespace default already exists 2025-06-04 14:15:53,841 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-06-04 14:15:53,962 - DEBUG - Output : cluster_project={} 2025-06-04 14:15:54,003 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api 2025-06-04 14:15:54,003 - DEBUG - Project uuid in Contrail is 7dc6db24-cf2b-4152-86d7-419fe1fbfa16 2025-06-04 14:15:54,003 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-06-04 14:15:54,124 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-06-04 14:15:54,124 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_NamespaceDatabaseList?namespace_uuid=dd7375f4-1cdb-42b8-af05-659c1198ea1d 2025-06-04 14:15:54,129 - INFO - Namespace default with uuid dd7375f4-1cdb-42b8-af05-659c1198ea1d found in kube manager 2025-06-04 14:15:54,129 - INFO - Namespace default verification passed 2025-06-04 14:15:54,132 - DEBUG - Service ctest-nginx-svc-89047507 not present 2025-06-04 14:15:54,133 - INFO - Creating service ctest-nginx-svc-89047507 2025-06-04 14:15:54,153 - INFO - Created Service ctest-nginx-svc-89047507 2025-06-04 14:15:54,161 - DEBUG - Pod ctest-nginx-pod-34726633 not present 2025-06-04 14:15:54,161 - INFO - Creating Pod ctest-nginx-pod-34726633 2025-06-04 14:15:54,219 - DEBUG - Node for Pod ctest-nginx-pod-34726633 not yet populated 2025-06-04 14:15:57,227 - DEBUG - Pod : ctest-nginx-pod-34726633 UUID is f892e1cf-72d2-47b1-a8c7-fcc778663ea2 2025-06-04 14:15:57,231 - DEBUG - Pod ctest-nginx-pod-19500668 not present 2025-06-04 14:15:57,231 - INFO - Creating Pod ctest-nginx-pod-19500668 2025-06-04 14:15:57,272 - DEBUG - Pod : ctest-nginx-pod-19500668 UUID is 3b64b73b-f53f-4355-935e-44221b9f7eb9 2025-06-04 14:15:57,533 - INFO - Created VN __public__, UUID :6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:57,838 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-06-04 14:15:57,853 - DEBUG - Requesting: http://10.0.0.38:8082/domain/88050c80-e5c4-4cc3-a3f1-b1ae9092f7cf 2025-06-04 14:15:57,872 - DEBUG - Requesting: http://10.0.0.38:8082/project/7dc6db24-cf2b-4152-86d7-419fe1fbfa16 2025-06-04 14:15:57,944 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:57,967 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:57,977 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/87cf843c-5c02-47b6-8821-5f29e459b0e4 2025-06-04 14:15:57,986 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/87cf843c-5c02-47b6-8821-5f29e459b0e4 2025-06-04 14:15:57,993 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/98662044-5aa7-4a49-a798-f139644178f0 2025-06-04 14:15:58,003 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-06-04 14:15:58,003 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:58,013 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/87cf843c-5c02-47b6-8821-5f29e459b0e4 2025-06-04 14:15:58,033 - INFO - Verified VN network id 13 for VN 6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:58,033 - INFO - Verifications in API Server for VN __public__ passed 2025-06-04 14:15:58,033 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:58,043 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/87cf843c-5c02-47b6-8821-5f29e459b0e4 2025-06-04 14:15:58,074 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/87cf843c-5c02-47b6-8821-5f29e459b0e4 2025-06-04 14:15:58,083 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/98662044-5aa7-4a49-a798-f139644178f0 2025-06-04 14:15:58,090 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-06-04 14:15:58,097 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '7729464008685405594', 'uuid-lslong': '11894528079430297191'}, 'enable': 'true', 'created': '2025-06-04T14:15:57', 'last-modified': '2025-06-04T14:15:57', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.318109'} 2025-06-04 14:15:58,097 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:15:58,104 - DEBUG - Route Targets: ['target:64512:8000007'] 2025-06-04 14:15:58,104 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed 2025-06-04 14:15:58,104 - DEBUG - ====Verifying policy data for __public__ in API_Server ====== 2025-06-04 14:15:58,105 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-06-04 14:15:58,111 - DEBUG - Requesting: http://10.0.0.38:8082/domain/88050c80-e5c4-4cc3-a3f1-b1ae9092f7cf 2025-06-04 14:15:58,120 - DEBUG - Requesting: http://10.0.0.38:8082/project/7dc6db24-cf2b-4152-86d7-419fe1fbfa16 2025-06-04 14:15:58,172 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:58,190 - DEBUG - =>VN __public__ has no policy to be verified 2025-06-04 14:15:58,190 - DEBUG - Verifying the vn in opserver 2025-06-04 14:15:58,190 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.38 2025-06-04 14:15:58,190 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks 2025-06-04 14:15:58,195 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'} 2025-06-04 14:15:58,195 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver 2025-06-04 14:15:58,196 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:15:58,206 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:15:58,216 - DEBUG - Do not have enough data to verify VN in agent 2025-06-04 14:15:58,216 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:15:58,224 - DEBUG - VRF ids for VN __public__: {} 2025-06-04 14:15:58,376 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server 2025-06-04 14:15:58,476 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-06-04 14:15:58,482 - DEBUG - Requesting: http://10.0.0.38:8082/domain/88050c80-e5c4-4cc3-a3f1-b1ae9092f7cf 2025-06-04 14:15:58,492 - DEBUG - Requesting: http://10.0.0.38:8082/project/7dc6db24-cf2b-4152-86d7-419fe1fbfa16 2025-06-04 14:15:58,536 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:58,547 - DEBUG - Requesting: http://10.0.0.38:8082/floating-ip-pool/4b446a73-a18d-44ad-8d5a-e0ec4c72715a 2025-06-04 14:15:58,552 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-06-04 14:15:58,556 - DEBUG - Requesting: http://10.0.0.38:8082/domain/88050c80-e5c4-4cc3-a3f1-b1ae9092f7cf 2025-06-04 14:15:58,563 - DEBUG - Requesting: http://10.0.0.38:8082/project/7dc6db24-cf2b-4152-86d7-419fe1fbfa16 2025-06-04 14:15:58,605 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/6b449290-3ddc-459a-a511-da44b15a2667 2025-06-04 14:15:58,615 - DEBUG - FIP Pool __fip_pool_public__ found in API Server 2025-06-04 14:15:58,615 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed 2025-06-04 14:15:58,622 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-06-04 14:15:58,629 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information 2025-06-04 14:15:58,629 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed 2025-06-04 14:15:58,633 - DEBUG - Ingress ctest-nginx-ingress-94381064 not present 2025-06-04 14:15:58,633 - INFO - Creating Ingress ctest-nginx-ingress-94381064 2025-06-04 14:15:58,640 - INFO - Created Ingress ctest-nginx-ingress-94381064 2025-06-04 14:15:58,642 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-94381064 2025-06-04 14:16:01,648 - DEBUG - For Ingress ctest-nginx-ingress-94381064, Cluster IP: 10.47.255.249, External IPs ['96.79.122.188'] 2025-06-04 14:16:01,648 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-94381064 2025-06-04 14:16:01,648 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=54198ad9-76cb-4201-99fd-d689de6c4a38 2025-06-04 14:16:01,659 - INFO - Ingress ctest-nginx-ingress-94381064 with uuid 54198ad9-76cb-4201-99fd-d689de6c4a38 found in kube manager 2025-06-04 14:16:01,664 - INFO - Ingress ctest-nginx-ingress-94381064 verification passed 2025-06-04 14:16:01,668 - DEBUG - Pod ctest-busybox-pod-42351170 not present 2025-06-04 14:16:01,669 - INFO - Creating Pod ctest-busybox-pod-42351170 2025-06-04 14:16:01,710 - DEBUG - Pod : ctest-busybox-pod-42351170 UUID is aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:01,719 - DEBUG - Pod ctest-nginx-pod-34726633 not in running state.Currently in Pending 2025-06-04 14:16:06,726 - INFO - Pod ctest-nginx-pod-34726633 is in running state.Got IP 10.47.255.251 2025-06-04 14:16:06,785 - DEBUG - Pod ctest-nginx-pod-34726633 has vmi 6df38f56-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:16:06,785 - INFO - Verified pod ctest-nginx-pod-34726633 in contrail-api 2025-06-04 14:16:06,786 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:16:06,799 - DEBUG - VMI 6df38f56-414e-11f0-8ea4-fa163eabd4a2 is active in agent 10.0.0.57 2025-06-04 14:16:06,799 - INFO - Verified Pod ctest-nginx-pod-34726633 in agent 10.0.0.57 2025-06-04 14:16:06,800 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=f892e1cf-72d2-47b1-a8c7-fcc778663ea2 2025-06-04 14:16:06,806 - INFO - Pod ctest-nginx-pod-34726633 with uuid f892e1cf-72d2-47b1-a8c7-fcc778663ea2 found in kube manager 2025-06-04 14:16:06,807 - INFO - Pod ctest-nginx-pod-34726633 verification passed 2025-06-04 14:16:06,922 - DEBUG - [Pod ctest-nginx-pod-34726633] Cmd: echo ctest-nginx-pod-34726633 > /usr/share/nginx/html/index.html, Output: 2025-06-04 14:16:06,928 - DEBUG - Pod ctest-nginx-pod-19500668 not in running state.Currently in Pending 2025-06-04 14:16:11,934 - INFO - Pod ctest-nginx-pod-19500668 is in running state.Got IP 10.47.255.250 2025-06-04 14:16:11,947 - DEBUG - Pod ctest-nginx-pod-19500668 has vmi 6f0e4a5c-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:16:11,947 - INFO - Verified pod ctest-nginx-pod-19500668 in contrail-api 2025-06-04 14:16:11,947 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:16:11,958 - DEBUG - VMI 6f0e4a5c-414e-11f0-8ea4-fa163eabd4a2 is active in agent 10.0.0.57 2025-06-04 14:16:11,958 - INFO - Verified Pod ctest-nginx-pod-19500668 in agent 10.0.0.57 2025-06-04 14:16:11,958 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=3b64b73b-f53f-4355-935e-44221b9f7eb9 2025-06-04 14:16:11,966 - INFO - Pod ctest-nginx-pod-19500668 with uuid 3b64b73b-f53f-4355-935e-44221b9f7eb9 found in kube manager 2025-06-04 14:16:11,966 - INFO - Pod ctest-nginx-pod-19500668 verification passed 2025-06-04 14:16:12,025 - DEBUG - [Pod ctest-nginx-pod-19500668] Cmd: echo ctest-nginx-pod-19500668 > /usr/share/nginx/html/index.html, Output: 2025-06-04 14:16:12,031 - INFO - Pod ctest-busybox-pod-42351170 is in running state.Got IP 10.47.255.248 2025-06-04 14:16:12,057 - DEBUG - Pod ctest-busybox-pod-42351170 has vmi 71adceae-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:16:12,057 - INFO - Verified pod ctest-busybox-pod-42351170 in contrail-api 2025-06-04 14:16:12,058 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:16:12,071 - DEBUG - VMI 71adceae-414e-11f0-8ea4-fa163eabd4a2 is active in agent 10.0.0.57 2025-06-04 14:16:12,071 - INFO - Verified Pod ctest-busybox-pod-42351170 in agent 10.0.0.57 2025-06-04 14:16:12,071 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:12,076 - INFO - Pod ctest-busybox-pod-42351170 with uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 found in kube manager 2025-06-04 14:16:12,077 - INFO - Pod ctest-busybox-pod-42351170 verification passed 2025-06-04 14:16:12,138 - DEBUG - [Pod ctest-busybox-pod-42351170] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-19500668 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-06-04 14:16:12,138 - DEBUG - [Pod Pod ctest-busybox-pod-42351170] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-06-04 14:16:12,138 - DEBUG - [Pod Pod ctest-busybox-pod-42351170] Cmd output: ctest-nginx-pod-19500668 Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout 2025-06-04 14:16:12,197 - DEBUG - [Pod ctest-busybox-pod-42351170] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-34726633 2025-06-04 14:16:12,197 - DEBUG - [Pod Pod ctest-busybox-pod-42351170] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed 2025-06-04 14:16:12,197 - DEBUG - [Pod Pod ctest-busybox-pod-42351170] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80) writing to stdout - 100% |********************************| 25 0:00:00 ETA written to stdout ctest-nginx-pod-34726633 2025-06-04 14:16:12,197 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-34726633': 1, 'ctest-nginx-pod-19500668': 1} 2025-06-04 14:16:12,198 - WARNING - Nothing to delete parallely 2025-06-04 14:16:12,198 - INFO - Deleting pod default:ctest-busybox-pod-42351170 2025-06-04 14:16:12,208 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:12,215 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:17,216 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:17,223 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:22,224 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:22,231 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:27,232 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:27,239 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:32,240 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:32,248 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:37,250 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:37,263 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:42,264 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:42,271 - WARNING - Pod uuid aa53ef5c-341f-487b-922c-5aa7903f0ae4 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:47,272 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=aa53ef5c-341f-487b-922c-5aa7903f0ae4 2025-06-04 14:16:47,279 - DEBUG - Pod aa53ef5c-341f-487b-922c-5aa7903f0ae4 is not in agent 10.0.0.57 VM list 2025-06-04 14:16:47,279 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:71adceae-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:16:47,286 - DEBUG - VMI 71adceae-414e-11f0-8ea4-fa163eabd4a2 is removed from agent 10.0.0.57 2025-06-04 14:16:47,286 - INFO - Verified that pod ctest-busybox-pod-42351170 is removed in agent 2025-06-04 14:16:47,286 - INFO - Deleting Ingress : ctest-nginx-ingress-94381064 2025-06-04 14:16:47,300 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=54198ad9-76cb-4201-99fd-d689de6c4a38 2025-06-04 14:16:47,305 - ERROR - Ingress ctest-nginx-ingress-94381064 with uuid 54198ad9-76cb-4201-99fd-d689de6c4a38 still found in kube manager 2025-06-04 14:16:48,307 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=54198ad9-76cb-4201-99fd-d689de6c4a38 2025-06-04 14:16:48,311 - INFO - Ingress ctest-nginx-ingress-94381064 with uuid 54198ad9-76cb-4201-99fd-d689de6c4a38 deleted successfully from kube manager 2025-06-04 14:16:48,311 - INFO - Verifications on Ingress %s cleanup passed 2025-06-04 14:16:48,311 - INFO - Deleting the FIP pool __fip_pool_public__ 2025-06-04 14:16:48,441 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__ 2025-06-04 14:16:48,448 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information 2025-06-04 14:16:48,448 - INFO - Deleting VN __public__ 2025-06-04 14:16:48,597 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/87cf843c-5c02-47b6-8821-5f29e459b0e4 2025-06-04 14:16:48,622 - DEBUG - Response Code: 404 2025-06-04 14:16:48,622 - DEBUG - Requesting: http://10.0.0.38:8082/domains 2025-06-04 14:16:48,636 - DEBUG - Requesting: http://10.0.0.38:8082/domain/88050c80-e5c4-4cc3-a3f1-b1ae9092f7cf 2025-06-04 14:16:48,650 - DEBUG - Requesting: http://10.0.0.38:8082/project/7dc6db24-cf2b-4152-86d7-419fe1fbfa16 2025-06-04 14:16:48,701 - INFO - Validated that VN __public__ is not found in API Server 2025-06-04 14:16:48,701 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__ 2025-06-04 14:16:48,711 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:16:48,727 - DEBUG - VN __public__ is not present in Agent 10.0.0.57 2025-06-04 14:16:48,727 - INFO - Validated that VN __public__ is not in any agent 2025-06-04 14:16:48,727 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:16:48,744 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__ 2025-06-04 14:16:48,750 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__ 2025-06-04 14:16:48,758 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info 2025-06-04 14:16:48,759 - INFO - Deleting pod default:ctest-nginx-pod-19500668 2025-06-04 14:16:48,789 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=3b64b73b-f53f-4355-935e-44221b9f7eb9 2025-06-04 14:16:48,804 - WARNING - Pod uuid 3b64b73b-f53f-4355-935e-44221b9f7eb9 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:53,805 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=3b64b73b-f53f-4355-935e-44221b9f7eb9 2025-06-04 14:16:53,812 - DEBUG - Pod 3b64b73b-f53f-4355-935e-44221b9f7eb9 is not in agent 10.0.0.57 VM list 2025-06-04 14:16:53,812 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6f0e4a5c-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:16:53,819 - DEBUG - VMI 6f0e4a5c-414e-11f0-8ea4-fa163eabd4a2 is removed from agent 10.0.0.57 2025-06-04 14:16:53,819 - INFO - Verified that pod ctest-nginx-pod-19500668 is removed in agent 2025-06-04 14:16:53,819 - INFO - Deleting pod default:ctest-nginx-pod-34726633 2025-06-04 14:16:53,827 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f892e1cf-72d2-47b1-a8c7-fcc778663ea2 2025-06-04 14:16:53,838 - WARNING - Pod uuid f892e1cf-72d2-47b1-a8c7-fcc778663ea2 is still seen in agent 10.0.0.57 VM list 2025-06-04 14:16:58,838 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f892e1cf-72d2-47b1-a8c7-fcc778663ea2 2025-06-04 14:16:58,851 - DEBUG - Pod f892e1cf-72d2-47b1-a8c7-fcc778663ea2 is not in agent 10.0.0.57 VM list 2025-06-04 14:16:58,852 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6df38f56-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:16:58,858 - DEBUG - VMI 6df38f56-414e-11f0-8ea4-fa163eabd4a2 is removed from agent 10.0.0.57 2025-06-04 14:16:58,858 - INFO - Verified that pod ctest-nginx-pod-34726633 is removed in agent 2025-06-04 14:16:58,859 - INFO - Deleting service : ctest-nginx-svc-89047507 2025-06-04 14:16:59,104 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AgentXmppConnectionStatusReq? 2025-06-04 14:16:59,112 - DEBUG - No XMPP flaps were noticed during the test 2025-06-04 14:16:59,113 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:06] 2025-06-04 14:16:59,113 - INFO - -------------------------------------------------------------------------------- 2025-06-04 14:17:05,608 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-06-04 14:17:05,768 - DEBUG - Output : cluster_project={} 2025-06-04 14:17:05,894 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN' 2025-06-04 14:17:06,084 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3 2025-06-04 14:17:06,158 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf' 2025-06-04 14:17:06,305 - DEBUG - Output : cluster_project={} 2025-06-04 14:17:21,561 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:21,616 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:21,693 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:26,739 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:26,829 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:26,865 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:27,013 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:27,050 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0, 2025-06-04 14:17:27,297 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AgentXmppConnectionStatusReq? 2025-06-04 14:17:29,892 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_ApplicationPolicySetReq? 2025-06-04 14:17:29,901 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AclReq? 2025-06-04 14:17:29,935 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AclReq? 2025-06-04 14:17:31,971 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AclReq? 2025-06-04 14:20:32,271 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_AgentXmppConnectionStatusReq? 2025-06-04 14:20:32,309 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=ac7c6bed-23d4-4d62-bc50-8dc020ca74be 2025-06-04 14:20:32,348 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=74547114-6f95-4b96-8a9c-e1c8a06bd3ae 2025-06-04 14:20:32,390 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=98b1f2c6-f611-453b-bac1-86a2d504c100 2025-06-04 14:20:32,404 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:20:32,408 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:20:32,429 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:20:32,433 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:20:32,450 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=d9fdc183-2171-4186-8312-42f3c7ce181b 2025-06-04 14:20:37,448 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=ac7c6bed-23d4-4d62-bc50-8dc020ca74be 2025-06-04 14:20:37,456 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:98adbbc2-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:20:37,467 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=74547114-6f95-4b96-8a9c-e1c8a06bd3ae 2025-06-04 14:20:37,477 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:993b73b8-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:20:37,525 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:20:37,527 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=98b1f2c6-f611-453b-bac1-86a2d504c100 2025-06-04 14:20:37,529 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:20:37,533 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=d9fdc183-2171-4186-8312-42f3c7ce181b 2025-06-04 14:20:37,536 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:20:37,538 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:20:37,573 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9a209d44-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:20:37,575 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9a9fe2e8-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:20:42,578 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:20:42,580 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:20:42,581 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:20:42,584 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:20:47,598 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:20:47,600 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:20:47,602 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:20:47,603 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:20:52,622 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:20:52,624 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:20:52,626 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:20:52,627 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:20:57,642 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:20:57,644 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:20:57,645 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:20:57,646 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:21:02,666 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:21:02,668 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:21:02,669 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:21:02,670 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:21:07,686 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=7cea8263-9d88-4272-90f6-89277b54f21c 2025-06-04 14:21:07,688 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=21aeeb0a-ac53-4f97-bbbc-86cb6d0e7883 2025-06-04 14:21:07,690 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=f9551374-5dfe-4ff3-9513-fffd5fc15c8f 2025-06-04 14:21:07,691 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_VmListReq?uuid=62b4b0c7-56bd-4fd8-86c6-9fc78697a718 2025-06-04 14:21:07,706 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9c3cd2d2-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:21:07,708 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9b3181da-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:21:07,711 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9cb94f42-414e-11f0-8ea4-fa163eabd4a2 2025-06-04 14:21:07,712 - DEBUG - Requesting: http://10.0.0.57:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9bbc8eb0-414e-11f0-8ea4-fa163eabd4a2