2025-08-11 18:35:50,687 - DEBUG - [10.0.0.130]: Running cmd : hostname
2025-08-11 18:35:51,037 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1366-1
2025-08-11 18:35:51,037 - DEBUG - [10.0.0.130]: Running cmd : hostname -f
2025-08-11 18:35:51,084 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1366-1
2025-08-11 18:35:51,085 - DEBUG - [10.0.0.130]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 18:35:51,247 - DEBUG - Output : NAMES
contrail_test_2FWJNzu0M
contrail-kubernetes-master-kubemanager-1
analytics_database-cassandra-1
analytics_database-nodemgr-1
analytics_database-query-engine-1
analytics_snmp-nodemgr-1
analytics_snmp-topology-1
analytics_snmp-snmp-collector-1
analytics_alarm-nodemgr-1
analytics_alarm-kafka-1
analytics_alarm-alarm-gen-1
analytics-nodemgr-1
analytics-api-1
analytics-collector-1
redis-redis-1
webui-web-1
webui-job-1
control-named-1
control-nodemgr-1
control-control-1
control-dns-1
config_database-cassandra-1
config_database-rabbitmq-1
config_database-zookeeper-1
config_database-nodemgr-1
config_api-devicemgr-1
config_api-nodemgr-1
config_api-dnsmasq-1
config_api-svcmonitor-1
config_api-api-1
config_api-schema-1
2025-08-11 18:35:51,248 - DEBUG - [10.0.0.130]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-11 18:35:51,282 - DEBUG - Output : 127.0.0.1/8
10.0.0.130/24
252.130.0.1/8
10.164.181.1/24
172.17.0.1/16
2025-08-11 18:35:51,283 - DEBUG - [10.0.0.130]: Running cmd : getent hosts 10.0.0.130 | head -n 1 | awk '{print $2}'
2025-08-11 18:35:51,377 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1366-1
2025-08-11 18:35:51,461 - DEBUG - [10.0.0.131]: Running cmd : hostname
2025-08-11 18:35:51,678 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1366-1
2025-08-11 18:35:51,679 - DEBUG - [10.0.0.131]: Running cmd : hostname -f
2025-08-11 18:35:51,704 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1366-1
2025-08-11 18:35:51,704 - DEBUG - [10.0.0.131]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-11 18:35:51,784 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-08-11 18:35:51,784 - DEBUG - [10.0.0.131]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-11 18:35:51,842 - DEBUG - Output : 10.0.0.131/24
2025-08-11 18:35:51,842 - DEBUG - [10.0.0.131]: Running cmd : getent hosts 10.0.0.131 | head -n 1 | awk '{print $2}'
2025-08-11 18:35:51,876 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1366-1
2025-08-11 18:35:51,956 - INFO - Using existing project ['default-domain', 'k8s-default'](85946e6e-5148-48c6-9b27-3ef85e842597)
2025-08-11 18:35:52,023 - INFO - Using existing project ['default-domain', 'k8s-default'](85946e6e-5148-48c6-9b27-3ef85e842597)
2025-08-11 18:35:52,188 - INFO - ================================================================================
2025-08-11 18:35:52,188 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-08-11 18:35:52,190 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-08-11 18:35:52,363 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 18:35:52,370 - DEBUG - Nothing to compare xmpp stats {'10.0.0.131': {'10.0.0.130': '0'}} with
2025-08-11 18:35:52,370 - INFO - Initial checks done. Running the testcase now
2025-08-11 18:35:52,371 - INFO -
2025-08-11 18:35:52,376 - INFO - Namespace default already exists
2025-08-11 18:35:52,376 - DEBUG - [10.0.0.130]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-08-11 18:35:52,517 - DEBUG - Output : cluster_project={}
2025-08-11 18:35:52,627 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-08-11 18:35:52,628 - DEBUG - Project uuid in Contrail is 85946e6e-5148-48c6-9b27-3ef85e842597
2025-08-11 18:35:52,628 - DEBUG - [10.0.0.130]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-08-11 18:35:52,757 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-08-11 18:35:52,757 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_NamespaceDatabaseList?namespace_uuid=234d9d16-a3be-4c98-9133-748b656cca2e
2025-08-11 18:35:52,762 - INFO - Namespace default with uuid 234d9d16-a3be-4c98-9133-748b656cca2e found in kube manager
2025-08-11 18:35:52,762 - INFO - Namespace default verification passed
2025-08-11 18:35:52,764 - DEBUG - Service ctest-nginx-svc-27636573 not present
2025-08-11 18:35:52,765 - INFO - Creating service ctest-nginx-svc-27636573
2025-08-11 18:35:52,804 - INFO - Created Service ctest-nginx-svc-27636573
2025-08-11 18:35:52,807 - DEBUG - Pod ctest-nginx-pod-18392857 not present
2025-08-11 18:35:52,807 - INFO - Creating Pod ctest-nginx-pod-18392857
2025-08-11 18:35:52,836 - DEBUG - Node for Pod ctest-nginx-pod-18392857 not yet populated
2025-08-11 18:35:55,843 - DEBUG - Pod : ctest-nginx-pod-18392857 UUID is cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57
2025-08-11 18:35:55,845 - DEBUG - Pod ctest-nginx-pod-84560248 not present
2025-08-11 18:35:55,846 - INFO - Creating Pod ctest-nginx-pod-84560248
2025-08-11 18:35:55,865 - DEBUG - Pod : ctest-nginx-pod-84560248 UUID is 3a293cc7-bbcc-42c2-b4a5-a35dd889f700
2025-08-11 18:35:56,085 - INFO - Created VN __public__, UUID :daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,413 - DEBUG - Requesting: http://10.0.0.130:8082/domains
2025-08-11 18:35:56,419 - DEBUG - Requesting: http://10.0.0.130:8082/domain/d5c63425-f374-4586-ae11-896bb191b2dc
2025-08-11 18:35:56,452 - DEBUG - Requesting: http://10.0.0.130:8082/project/85946e6e-5148-48c6-9b27-3ef85e842597
2025-08-11 18:35:56,515 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,529 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,539 - DEBUG - Requesting: http://10.0.0.130:8082/routing-instance/898428ec-04b5-416f-b790-b1c2ab149cf4
2025-08-11 18:35:56,555 - DEBUG - Requesting: http://10.0.0.130:8082/routing-instance/898428ec-04b5-416f-b790-b1c2ab149cf4
2025-08-11 18:35:56,568 - DEBUG - Requesting: http://10.0.0.130:8082/route-target/aaa46bdc-c8ca-4882-b9fd-c39e3b0c2a14
2025-08-11 18:35:56,578 - DEBUG - Route Targets: ['target:64512:8000007']
2025-08-11 18:35:56,578 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,587 - DEBUG - Requesting: http://10.0.0.130:8082/routing-instance/898428ec-04b5-416f-b790-b1c2ab149cf4
2025-08-11 18:35:56,599 - INFO - Verified VN network id 14 for VN daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,599 - INFO - Verifications in API Server for VN __public__ passed
2025-08-11 18:35:56,599 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,608 - DEBUG - Requesting: http://10.0.0.130:8082/routing-instance/898428ec-04b5-416f-b790-b1c2ab149cf4
2025-08-11 18:35:56,613 - DEBUG - Requesting: http://10.0.0.130:8082/routing-instance/898428ec-04b5-416f-b790-b1c2ab149cf4
2025-08-11 18:35:56,619 - DEBUG - Requesting: http://10.0.0.130:8082/route-target/aaa46bdc-c8ca-4882-b9fd-c39e3b0c2a14
2025-08-11 18:35:56,626 - DEBUG - Requesting: http://10.0.0.130:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-08-11 18:35:56,635 - DEBUG - Control-node 10.0.0.130 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15777106673346629603', 'uuid-lslong': '11682592095003033544'}, 'enable': 'true', 'created': '2025-08-11T18:35:55', 'last-modified': '2025-08-11T18:35:56', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.332229'}
2025-08-11 18:35:56,635 - DEBUG - Requesting: http://10.0.0.130:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:35:56,642 - DEBUG - Route Targets: ['target:64512:8000007']
2025-08-11 18:35:56,642 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-08-11 18:35:56,643 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-08-11 18:35:56,643 - DEBUG - Requesting: http://10.0.0.130:8082/domains
2025-08-11 18:35:56,660 - DEBUG - Requesting: http://10.0.0.130:8082/domain/d5c63425-f374-4586-ae11-896bb191b2dc
2025-08-11 18:35:56,696 - DEBUG - Requesting: http://10.0.0.130:8082/project/85946e6e-5148-48c6-9b27-3ef85e842597
2025-08-11 18:35:56,756 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:56,777 - DEBUG - =>VN __public__ has no policy to be verified
2025-08-11 18:35:56,777 - DEBUG - Verifying the vn in opserver
2025-08-11 18:35:56,777 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.130
2025-08-11 18:35:56,777 - DEBUG - Requesting: http://10.0.0.130:8081/analytics/uves/virtual-networks
2025-08-11 18:35:56,792 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.130:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-08-11 18:35:56,792 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-08-11 18:35:56,793 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:35:56,800 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:35:56,806 - DEBUG - Do not have enough data to verify VN in agent
2025-08-11 18:35:56,806 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:35:56,812 - DEBUG - VRF ids for VN __public__: {}
2025-08-11 18:35:56,907 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-08-11 18:35:57,011 - DEBUG - Requesting: http://10.0.0.130:8082/domains
2025-08-11 18:35:57,016 - DEBUG - Requesting: http://10.0.0.130:8082/domain/d5c63425-f374-4586-ae11-896bb191b2dc
2025-08-11 18:35:57,023 - DEBUG - Requesting: http://10.0.0.130:8082/project/85946e6e-5148-48c6-9b27-3ef85e842597
2025-08-11 18:35:57,068 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:57,079 - DEBUG - Requesting: http://10.0.0.130:8082/floating-ip-pool/51c75c52-33d1-4108-a595-a6a4a952ac19
2025-08-11 18:35:57,089 - DEBUG - Requesting: http://10.0.0.130:8082/domains
2025-08-11 18:35:57,093 - DEBUG - Requesting: http://10.0.0.130:8082/domain/d5c63425-f374-4586-ae11-896bb191b2dc
2025-08-11 18:35:57,100 - DEBUG - Requesting: http://10.0.0.130:8082/project/85946e6e-5148-48c6-9b27-3ef85e842597
2025-08-11 18:35:57,154 - DEBUG - Requesting: http://10.0.0.130:8082/virtual-network/daf38aed-c045-4fe3-a220-e79d06713fc8
2025-08-11 18:35:57,163 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-08-11 18:35:57,163 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-08-11 18:35:57,170 - DEBUG - Requesting: http://10.0.0.130:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-08-11 18:35:57,176 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-08-11 18:35:57,176 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-08-11 18:35:57,179 - DEBUG - Ingress ctest-nginx-ingress-38879009 not present
2025-08-11 18:35:57,179 - INFO - Creating Ingress ctest-nginx-ingress-38879009
2025-08-11 18:35:57,188 - INFO - Created Ingress ctest-nginx-ingress-38879009
2025-08-11 18:35:57,190 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-38879009
2025-08-11 18:36:00,196 - DEBUG - For Ingress ctest-nginx-ingress-38879009, Cluster IP: 10.47.255.249, External IPs ['186.208.137.188']
2025-08-11 18:36:00,196 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-38879009
2025-08-11 18:36:00,196 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_LoadbalancerDatabaseList?x=3e78dc2f-ab3f-4527-9dc9-8b11cf2e0dc7
2025-08-11 18:36:00,201 - INFO - Ingress ctest-nginx-ingress-38879009 with uuid 3e78dc2f-ab3f-4527-9dc9-8b11cf2e0dc7 found in kube manager
2025-08-11 18:36:00,201 - INFO - Ingress ctest-nginx-ingress-38879009 verification passed
2025-08-11 18:36:00,204 - DEBUG - Pod ctest-busybox-pod-52085183 not present
2025-08-11 18:36:00,204 - INFO - Creating Pod ctest-busybox-pod-52085183
2025-08-11 18:36:00,226 - DEBUG - Node for Pod ctest-busybox-pod-52085183 not yet populated
2025-08-11 18:36:03,234 - DEBUG - Pod : ctest-busybox-pod-52085183 UUID is 82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:03,241 - DEBUG - Pod ctest-nginx-pod-18392857 not in running state.Currently in Pending
2025-08-11 18:36:08,248 - INFO - Pod ctest-nginx-pod-18392857 is in running state.Got IP 10.47.255.251
2025-08-11 18:36:08,263 - DEBUG - Pod ctest-nginx-pod-18392857 has vmi 03a9e932-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:36:08,263 - INFO - Verified pod ctest-nginx-pod-18392857 in contrail-api
2025-08-11 18:36:08,263 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:36:08,275 - DEBUG - VMI 03a9e932-76e2-11f0-a165-fa163ef78b47 is active in agent 10.0.0.131
2025-08-11 18:36:08,275 - INFO - Verified Pod ctest-nginx-pod-18392857 in agent 10.0.0.131
2025-08-11 18:36:08,275 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_VirtualMachineDatabaseList?x=cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57
2025-08-11 18:36:08,279 - INFO - Pod ctest-nginx-pod-18392857 with uuid cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57 found in kube manager
2025-08-11 18:36:08,279 - INFO - Pod ctest-nginx-pod-18392857 verification passed
2025-08-11 18:36:08,346 - DEBUG - [Pod ctest-nginx-pod-18392857] Cmd: echo ctest-nginx-pod-18392857 > /usr/share/nginx/html/index.html, Output:
2025-08-11 18:36:08,352 - INFO - Pod ctest-nginx-pod-84560248 is in running state.Got IP 10.47.255.250
2025-08-11 18:36:08,408 - DEBUG - Pod ctest-nginx-pod-84560248 has vmi 04a1f56e-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:36:08,408 - INFO - Verified pod ctest-nginx-pod-84560248 in contrail-api
2025-08-11 18:36:08,408 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:36:08,418 - DEBUG - VMI 04a1f56e-76e2-11f0-a165-fa163ef78b47 is active in agent 10.0.0.131
2025-08-11 18:36:08,418 - INFO - Verified Pod ctest-nginx-pod-84560248 in agent 10.0.0.131
2025-08-11 18:36:08,418 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_VirtualMachineDatabaseList?x=3a293cc7-bbcc-42c2-b4a5-a35dd889f700
2025-08-11 18:36:08,422 - INFO - Pod ctest-nginx-pod-84560248 with uuid 3a293cc7-bbcc-42c2-b4a5-a35dd889f700 found in kube manager
2025-08-11 18:36:08,422 - INFO - Pod ctest-nginx-pod-84560248 verification passed
2025-08-11 18:36:08,475 - DEBUG - [Pod ctest-nginx-pod-84560248] Cmd: echo ctest-nginx-pod-84560248 > /usr/share/nginx/html/index.html, Output:
2025-08-11 18:36:08,484 - DEBUG - Pod ctest-busybox-pod-52085183 not in running state.Currently in Pending
2025-08-11 18:36:13,490 - DEBUG - Pod ctest-busybox-pod-52085183 not in running state.Currently in Pending
2025-08-11 18:36:18,497 - INFO - Pod ctest-busybox-pod-52085183 is in running state.Got IP 10.47.255.248
2025-08-11 18:36:18,510 - DEBUG - Pod ctest-busybox-pod-52085183 has vmi 073ba7b6-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:36:18,510 - INFO - Verified pod ctest-busybox-pod-52085183 in contrail-api
2025-08-11 18:36:18,511 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:36:18,523 - DEBUG - VMI 073ba7b6-76e2-11f0-a165-fa163ef78b47 is active in agent 10.0.0.131
2025-08-11 18:36:18,523 - INFO - Verified Pod ctest-busybox-pod-52085183 in agent 10.0.0.131
2025-08-11 18:36:18,523 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_VirtualMachineDatabaseList?x=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:18,537 - INFO - Pod ctest-busybox-pod-52085183 with uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f found in kube manager
2025-08-11 18:36:18,538 - INFO - Pod ctest-busybox-pod-52085183 verification passed
2025-08-11 18:36:19,127 - DEBUG - [Pod ctest-busybox-pod-52085183] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-84560248
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-11 18:36:19,128 - DEBUG - [Pod Pod ctest-busybox-pod-52085183] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-08-11 18:36:19,128 - DEBUG - [Pod Pod ctest-busybox-pod-52085183] Cmd output: ctest-nginx-pod-84560248
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-11 18:36:19,174 - DEBUG - [Pod ctest-busybox-pod-52085183] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-18392857
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-11 18:36:19,174 - DEBUG - [Pod Pod ctest-busybox-pod-52085183] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-08-11 18:36:19,174 - DEBUG - [Pod Pod ctest-busybox-pod-52085183] Cmd output: ctest-nginx-pod-18392857
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-11 18:36:19,174 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-18392857': 1, 'ctest-nginx-pod-84560248': 1}
2025-08-11 18:36:19,174 - WARNING - Nothing to delete parallely
2025-08-11 18:36:19,175 - INFO - Deleting pod default:ctest-busybox-pod-52085183
2025-08-11 18:36:19,184 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:19,193 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:24,194 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:24,201 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:29,202 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:29,209 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:34,209 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:34,217 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:39,217 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:39,224 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:44,225 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:44,232 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:49,232 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:49,239 - WARNING - Pod uuid 82a4fd65-d734-4808-9d62-e6cda43ff75f is still seen in agent 10.0.0.131 VM list
2025-08-11 18:36:54,239 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=82a4fd65-d734-4808-9d62-e6cda43ff75f
2025-08-11 18:36:54,246 - DEBUG - Pod 82a4fd65-d734-4808-9d62-e6cda43ff75f is not in agent 10.0.0.131 VM list
2025-08-11 18:36:54,246 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:073ba7b6-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:36:54,252 - DEBUG - VMI 073ba7b6-76e2-11f0-a165-fa163ef78b47 is removed from agent 10.0.0.131
2025-08-11 18:36:54,252 - INFO - Verified that pod ctest-busybox-pod-52085183 is removed in agent
2025-08-11 18:36:54,252 - INFO - Deleting Ingress : ctest-nginx-ingress-38879009
2025-08-11 18:36:54,257 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_LoadbalancerDatabaseList?x=3e78dc2f-ab3f-4527-9dc9-8b11cf2e0dc7
2025-08-11 18:36:54,263 - ERROR - Ingress ctest-nginx-ingress-38879009 with uuid 3e78dc2f-ab3f-4527-9dc9-8b11cf2e0dc7 still found in kube manager
2025-08-11 18:36:55,264 - DEBUG - Requesting: http://10.0.0.130:8108/Snh_LoadbalancerDatabaseList?x=3e78dc2f-ab3f-4527-9dc9-8b11cf2e0dc7
2025-08-11 18:36:55,269 - INFO - Ingress ctest-nginx-ingress-38879009 with uuid 3e78dc2f-ab3f-4527-9dc9-8b11cf2e0dc7 deleted successfully from kube manager
2025-08-11 18:36:55,269 - INFO - Verifications on Ingress %s cleanup passed
2025-08-11 18:36:55,269 - INFO - Deleting the FIP pool __fip_pool_public__
2025-08-11 18:36:55,488 - DEBUG - Requesting: http://10.0.0.130:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-08-11 18:36:55,494 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-08-11 18:36:55,494 - INFO - Deleting VN __public__
2025-08-11 18:36:55,692 - DEBUG - Requesting: http://10.0.0.130:8082/routing-instance/898428ec-04b5-416f-b790-b1c2ab149cf4
2025-08-11 18:36:55,698 - DEBUG - Response Code: 404
2025-08-11 18:36:55,699 - DEBUG - Requesting: http://10.0.0.130:8082/domains
2025-08-11 18:36:55,703 - DEBUG - Requesting: http://10.0.0.130:8082/domain/d5c63425-f374-4586-ae11-896bb191b2dc
2025-08-11 18:36:55,725 - DEBUG - Requesting: http://10.0.0.130:8082/project/85946e6e-5148-48c6-9b27-3ef85e842597
2025-08-11 18:36:55,783 - INFO - Validated that VN __public__ is not found in API Server
2025-08-11 18:36:55,783 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-08-11 18:36:55,790 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:36:55,799 - DEBUG - VN __public__ is not present in Agent 10.0.0.131
2025-08-11 18:36:55,799 - INFO - Validated that VN __public__ is not in any agent
2025-08-11 18:36:55,799 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:36:55,809 - DEBUG - Requesting: http://10.0.0.130:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-08-11 18:36:55,817 - DEBUG - Requesting: http://10.0.0.130:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-08-11 18:36:55,831 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-08-11 18:36:55,832 - INFO - Deleting pod default:ctest-nginx-pod-84560248
2025-08-11 18:36:55,846 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=3a293cc7-bbcc-42c2-b4a5-a35dd889f700
2025-08-11 18:36:55,855 - WARNING - Pod uuid 3a293cc7-bbcc-42c2-b4a5-a35dd889f700 is still seen in agent 10.0.0.131 VM list
2025-08-11 18:37:00,856 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=3a293cc7-bbcc-42c2-b4a5-a35dd889f700
2025-08-11 18:37:00,862 - DEBUG - Pod 3a293cc7-bbcc-42c2-b4a5-a35dd889f700 is not in agent 10.0.0.131 VM list
2025-08-11 18:37:00,863 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:04a1f56e-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:37:00,868 - DEBUG - VMI 04a1f56e-76e2-11f0-a165-fa163ef78b47 is removed from agent 10.0.0.131
2025-08-11 18:37:00,868 - INFO - Verified that pod ctest-nginx-pod-84560248 is removed in agent
2025-08-11 18:37:00,868 - INFO - Deleting pod default:ctest-nginx-pod-18392857
2025-08-11 18:37:00,878 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57
2025-08-11 18:37:00,884 - WARNING - Pod uuid cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57 is still seen in agent 10.0.0.131 VM list
2025-08-11 18:37:05,886 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57
2025-08-11 18:37:05,893 - DEBUG - Pod cb68ea2a-a9d5-42f5-ace7-1efeb4f56b57 is not in agent 10.0.0.131 VM list
2025-08-11 18:37:05,893 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:03a9e932-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:37:05,898 - DEBUG - VMI 03a9e932-76e2-11f0-a165-fa163ef78b47 is removed from agent 10.0.0.131
2025-08-11 18:37:05,899 - INFO - Verified that pod ctest-nginx-pod-18392857 is removed in agent
2025-08-11 18:37:05,899 - INFO - Deleting service : ctest-nginx-svc-27636573
2025-08-11 18:37:06,079 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 18:37:06,085 - DEBUG - No XMPP flaps were noticed during the test
2025-08-11 18:37:06,085 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:14]
2025-08-11 18:37:06,086 - INFO - --------------------------------------------------------------------------------
2025-08-11 18:37:12,438 - DEBUG - [10.0.0.130]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-08-11 18:37:12,559 - DEBUG - Output : cluster_project={}
2025-08-11 18:37:12,597 - DEBUG - [10.0.0.130]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-08-11 18:37:12,728 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-08-11 18:37:12,752 - DEBUG - [10.0.0.130]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-08-11 18:37:12,939 - DEBUG - Output : cluster_project={}
2025-08-11 18:37:28,096 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:28,133 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:28,214 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:28,288 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:33,328 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:33,362 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:33,439 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:33,516 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-11 18:37:33,791 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 18:37:36,465 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_ApplicationPolicySetReq?
2025-08-11 18:37:36,480 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AclReq?
2025-08-11 18:37:36,576 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AclReq?
2025-08-11 18:37:38,607 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AclReq?
2025-08-11 18:40:38,671 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-11 18:40:38,705 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=2951c55a-6e09-4e20-8305-309c473ac2ae
2025-08-11 18:40:38,735 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d5f8ae9b-b776-4d0d-9420-272670bcd997
2025-08-11 18:40:38,740 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:40:38,746 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=21135c68-6b76-4e99-acf0-45b56af20404
2025-08-11 18:40:38,750 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:40:38,757 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=8cc0fc1f-0d5d-4b6b-821a-7abed2c7f620
2025-08-11 18:40:38,764 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:40:38,770 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:40:43,754 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=2951c55a-6e09-4e20-8305-309c473ac2ae
2025-08-11 18:40:43,762 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3308e0b6-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:40:43,807 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:40:43,810 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:40:43,812 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=8cc0fc1f-0d5d-4b6b-821a-7abed2c7f620
2025-08-11 18:40:43,814 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:40:43,815 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=21135c68-6b76-4e99-acf0-45b56af20404
2025-08-11 18:40:43,817 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d5f8ae9b-b776-4d0d-9420-272670bcd997
2025-08-11 18:40:43,820 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:40:43,838 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3373f3b0-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:40:43,842 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3418c052-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:40:43,844 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:34899af2-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:40:48,820 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:40:48,844 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:40:48,845 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:40:48,847 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:40:53,829 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:40:53,857 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:40:53,860 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:40:53,861 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:40:58,837 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:40:58,873 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:40:58,875 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:40:58,877 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:41:03,845 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:41:03,888 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:41:03,890 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:41:03,891 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:41:08,854 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:41:08,903 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:41:08,905 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:41:08,906 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:41:13,862 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=4b79c71e-5de1-4641-981d-2d4282adc37a
2025-08-11 18:41:13,870 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:350e9a22-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:41:13,917 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=12249190-bc76-48d6-9b20-d2357a82e302
2025-08-11 18:41:13,920 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=d7999f9e-5965-4d1f-ac90-5e22c4b7c2c7
2025-08-11 18:41:13,921 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_VmListReq?uuid=fa801520-7b8d-4e95-ac52-e941605671eb
2025-08-11 18:41:13,932 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:357fe812-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:41:13,935 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:35f4b44e-76e2-11f0-a165-fa163ef78b47
2025-08-11 18:41:13,937 - DEBUG - Requesting: http://10.0.0.131:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:366a2efe-76e2-11f0-a165-fa163ef78b47