2025-06-19 17:16:56,986 - DEBUG - [10.0.0.65]: Running cmd : hostname
2025-06-19 17:16:57,260 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1286-1
2025-06-19 17:16:57,260 - DEBUG - [10.0.0.65]: Running cmd : hostname -f
2025-06-19 17:16:57,294 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1286-1
2025-06-19 17:16:57,295 - DEBUG - [10.0.0.65]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-06-19 17:16:57,398 - DEBUG - Output : NAMES
contrail_test_VMh3m1uBC
contrail-kubernetes-master-kubemanager-1
analytics_snmp-topology-1
analytics_snmp-nodemgr-1
analytics_snmp-snmp-collector-1
analytics_alarm-nodemgr-1
analytics_alarm-kafka-1
analytics_alarm-alarm-gen-1
analytics-collector-1
analytics-api-1
analytics-nodemgr-1
analytics_database-query-engine-1
analytics_database-nodemgr-1
analytics_database-cassandra-1
redis-redis-1
webui-web-1
webui-job-1
control-named-1
control-dns-1
control-control-1
control-nodemgr-1
config_database-cassandra-1
config_database-rabbitmq-1
config_database-nodemgr-1
config_database-zookeeper-1
config_api-devicemgr-1
config_api-nodemgr-1
config_api-dnsmasq-1
config_api-api-1
config_api-svcmonitor-1
config_api-schema-1
2025-06-19 17:16:57,399 - DEBUG - [10.0.0.65]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-06-19 17:16:57,432 - DEBUG - Output : 10.0.0.65/24
172.17.0.1/16
2025-06-19 17:16:57,433 - DEBUG - [10.0.0.65]: Running cmd : getent hosts 10.0.0.65 | head -n 1 | awk '{print $2}'
2025-06-19 17:16:57,467 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1286-1
2025-06-19 17:16:57,568 - DEBUG - [10.0.0.27]: Running cmd : hostname
2025-06-19 17:16:57,789 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1286-1
2025-06-19 17:16:57,789 - DEBUG - [10.0.0.27]: Running cmd : hostname -f
2025-06-19 17:16:57,824 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1286-1
2025-06-19 17:16:57,824 - DEBUG - [10.0.0.27]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-06-19 17:16:57,913 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-06-19 17:16:57,914 - DEBUG - [10.0.0.27]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-06-19 17:16:57,948 - DEBUG - Output : 10.0.0.27/24
2025-06-19 17:16:57,948 - DEBUG - [10.0.0.27]: Running cmd : getent hosts 10.0.0.27 | head -n 1 | awk '{print $2}'
2025-06-19 17:16:57,983 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1286-1
2025-06-19 17:16:58,084 - INFO - Using existing project ['default-domain', 'k8s-default'](63c221f2-a419-4998-ab2d-4cbb7feb74f2)
2025-06-19 17:16:58,204 - INFO - Using existing project ['default-domain', 'k8s-default'](63c221f2-a419-4998-ab2d-4cbb7feb74f2)
2025-06-19 17:16:58,354 - INFO - ================================================================================
2025-06-19 17:16:58,354 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-06-19 17:16:58,355 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-06-19 17:16:58,563 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AgentXmppConnectionStatusReq?
2025-06-19 17:16:58,571 - DEBUG - Nothing to compare xmpp stats {'10.0.0.27': {'10.0.0.65': '0'}} with
2025-06-19 17:16:58,571 - INFO - Initial checks done. Running the testcase now
2025-06-19 17:16:58,571 - INFO -
2025-06-19 17:16:58,575 - INFO - Namespace default already exists
2025-06-19 17:16:58,576 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-06-19 17:16:58,694 - DEBUG - Output : cluster_project={}
2025-06-19 17:16:58,749 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-06-19 17:16:58,749 - DEBUG - Project uuid in Contrail is 63c221f2-a419-4998-ab2d-4cbb7feb74f2
2025-06-19 17:16:58,750 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-06-19 17:16:58,889 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-06-19 17:16:58,889 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_NamespaceDatabaseList?namespace_uuid=43eaaf72-d79e-4dda-a3a6-af1ee92ec22b
2025-06-19 17:16:58,894 - INFO - Namespace default with uuid 43eaaf72-d79e-4dda-a3a6-af1ee92ec22b found in kube manager
2025-06-19 17:16:58,894 - INFO - Namespace default verification passed
2025-06-19 17:16:58,897 - DEBUG - Service ctest-nginx-svc-98525832 not present
2025-06-19 17:16:58,898 - INFO - Creating service ctest-nginx-svc-98525832
2025-06-19 17:16:58,916 - INFO - Created Service ctest-nginx-svc-98525832
2025-06-19 17:16:58,933 - DEBUG - Pod ctest-nginx-pod-30777511 not present
2025-06-19 17:16:58,934 - INFO - Creating Pod ctest-nginx-pod-30777511
2025-06-19 17:16:59,009 - DEBUG - Pod : ctest-nginx-pod-30777511 UUID is 3ac818a1-e66f-488b-9094-2046e41b7179
2025-06-19 17:16:59,013 - DEBUG - Pod ctest-nginx-pod-29742453 not present
2025-06-19 17:16:59,013 - INFO - Creating Pod ctest-nginx-pod-29742453
2025-06-19 17:16:59,049 - DEBUG - Pod : ctest-nginx-pod-29742453 UUID is 6ce4b802-4157-47d1-b514-caeec194f694
2025-06-19 17:16:59,346 - INFO - Created VN __public__, UUID :f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:16:59,659 - DEBUG - Requesting: http://10.0.0.65:8082/domains
2025-06-19 17:16:59,677 - DEBUG - Requesting: http://10.0.0.65:8082/domain/6a247229-626e-4326-88a5-1fdd19f4d2d6
2025-06-19 17:16:59,728 - DEBUG - Requesting: http://10.0.0.65:8082/project/63c221f2-a419-4998-ab2d-4cbb7feb74f2
2025-06-19 17:16:59,870 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:16:59,912 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:16:59,942 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/2d9dd996-6643-4162-8ca3-792cef921049
2025-06-19 17:16:59,960 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/2d9dd996-6643-4162-8ca3-792cef921049
2025-06-19 17:16:59,987 - DEBUG - Requesting: http://10.0.0.65:8082/route-target/ba75cb8c-8934-447e-af03-a5835effab50
2025-06-19 17:17:00,003 - DEBUG - Route Targets: ['target:64512:8000007']
2025-06-19 17:17:00,003 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:17:00,058 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/2d9dd996-6643-4162-8ca3-792cef921049
2025-06-19 17:17:00,119 - INFO - Verified VN network id 13 for VN f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:17:00,119 - INFO - Verifications in API Server for VN __public__ passed
2025-06-19 17:17:00,119 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:17:00,133 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/2d9dd996-6643-4162-8ca3-792cef921049
2025-06-19 17:17:00,171 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/2d9dd996-6643-4162-8ca3-792cef921049
2025-06-19 17:17:00,186 - DEBUG - Requesting: http://10.0.0.65:8082/route-target/ba75cb8c-8934-447e-af03-a5835effab50
2025-06-19 17:17:00,221 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-06-19 17:17:00,240 - DEBUG - Control-node 10.0.0.65 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17492905802283895168', 'uuid-lslong': '13573878743875077075'}, 'enable': 'true', 'created': '2025-06-19T17:16:59', 'last-modified': '2025-06-19T17:16:59', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.651965'}
2025-06-19 17:17:00,240 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:00,258 - DEBUG - Route Targets: ['target:64512:8000007']
2025-06-19 17:17:00,259 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-06-19 17:17:00,259 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-06-19 17:17:00,259 - DEBUG - Requesting: http://10.0.0.65:8082/domains
2025-06-19 17:17:00,283 - DEBUG - Requesting: http://10.0.0.65:8082/domain/6a247229-626e-4326-88a5-1fdd19f4d2d6
2025-06-19 17:17:00,299 - DEBUG - Requesting: http://10.0.0.65:8082/project/63c221f2-a419-4998-ab2d-4cbb7feb74f2
2025-06-19 17:17:00,398 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:17:00,440 - DEBUG - =>VN __public__ has no policy to be verified
2025-06-19 17:17:00,440 - DEBUG - Verifying the vn in opserver
2025-06-19 17:17:00,441 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.65
2025-06-19 17:17:00,441 - DEBUG - Requesting: http://10.0.0.65:8081/analytics/uves/virtual-networks
2025-06-19 17:17:00,447 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.65:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-06-19 17:17:00,447 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-06-19 17:17:00,447 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:00,455 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:00,461 - DEBUG - Do not have enough data to verify VN in agent
2025-06-19 17:17:00,462 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:00,469 - DEBUG - VRF ids for VN __public__: {}
2025-06-19 17:17:00,595 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-06-19 17:17:00,873 - DEBUG - Requesting: http://10.0.0.65:8082/domains
2025-06-19 17:17:00,889 - DEBUG - Requesting: http://10.0.0.65:8082/domain/6a247229-626e-4326-88a5-1fdd19f4d2d6
2025-06-19 17:17:00,909 - DEBUG - Requesting: http://10.0.0.65:8082/project/63c221f2-a419-4998-ab2d-4cbb7feb74f2
2025-06-19 17:17:00,977 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:17:01,009 - DEBUG - Requesting: http://10.0.0.65:8082/floating-ip-pool/9fce376a-703a-4c76-b826-f18bcc55f4a2
2025-06-19 17:17:01,030 - DEBUG - Requesting: http://10.0.0.65:8082/domains
2025-06-19 17:17:01,041 - DEBUG - Requesting: http://10.0.0.65:8082/domain/6a247229-626e-4326-88a5-1fdd19f4d2d6
2025-06-19 17:17:01,055 - DEBUG - Requesting: http://10.0.0.65:8082/project/63c221f2-a419-4998-ab2d-4cbb7feb74f2
2025-06-19 17:17:01,131 - DEBUG - Requesting: http://10.0.0.65:8082/virtual-network/f2c34925-5280-4980-bc60-1accd0e643d3
2025-06-19 17:17:01,145 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-06-19 17:17:01,145 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-06-19 17:17:01,176 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-06-19 17:17:01,190 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-06-19 17:17:01,190 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-06-19 17:17:01,195 - DEBUG - Ingress ctest-nginx-ingress-36501067 not present
2025-06-19 17:17:01,195 - INFO - Creating Ingress ctest-nginx-ingress-36501067
2025-06-19 17:17:01,204 - INFO - Created Ingress ctest-nginx-ingress-36501067
2025-06-19 17:17:01,209 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-36501067
2025-06-19 17:17:04,224 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-36501067
2025-06-19 17:17:07,232 - DEBUG - For Ingress ctest-nginx-ingress-36501067, Cluster IP: 10.47.255.249, External IPs ['11.116.134.188']
2025-06-19 17:17:07,232 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-36501067
2025-06-19 17:17:07,232 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=62205333-9487-4459-a986-5a79ffc3960c
2025-06-19 17:17:07,241 - INFO - Ingress ctest-nginx-ingress-36501067 with uuid 62205333-9487-4459-a986-5a79ffc3960c found in kube manager
2025-06-19 17:17:07,241 - INFO - Ingress ctest-nginx-ingress-36501067 verification passed
2025-06-19 17:17:07,248 - DEBUG - Pod ctest-busybox-pod-80609421 not present
2025-06-19 17:17:07,248 - INFO - Creating Pod ctest-busybox-pod-80609421
2025-06-19 17:17:07,310 - DEBUG - Pod : ctest-busybox-pod-80609421 UUID is a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:07,318 - DEBUG - Pod ctest-nginx-pod-30777511 not in running state.Currently in Pending
2025-06-19 17:17:12,324 - DEBUG - Pod ctest-nginx-pod-30777511 not in running state.Currently in Pending
2025-06-19 17:17:17,332 - INFO - Pod ctest-nginx-pod-30777511 is in running state.Got IP 10.47.255.251
2025-06-19 17:17:17,352 - DEBUG - Pod ctest-nginx-pod-30777511 has vmi 36ae0d0c-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:17:17,353 - INFO - Verified pod ctest-nginx-pod-30777511 in contrail-api
2025-06-19 17:17:17,353 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:17:17,368 - DEBUG - VMI 36ae0d0c-4d31-11f0-8532-fa163ede8ab3 is active in agent 10.0.0.27
2025-06-19 17:17:17,368 - INFO - Verified Pod ctest-nginx-pod-30777511 in agent 10.0.0.27
2025-06-19 17:17:17,369 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_VirtualMachineDatabaseList?x=3ac818a1-e66f-488b-9094-2046e41b7179
2025-06-19 17:17:17,374 - INFO - Pod ctest-nginx-pod-30777511 with uuid 3ac818a1-e66f-488b-9094-2046e41b7179 found in kube manager
2025-06-19 17:17:17,374 - INFO - Pod ctest-nginx-pod-30777511 verification passed
2025-06-19 17:17:17,445 - DEBUG - [Pod ctest-nginx-pod-30777511] Cmd: echo ctest-nginx-pod-30777511 > /usr/share/nginx/html/index.html, Output:
2025-06-19 17:17:17,452 - INFO - Pod ctest-nginx-pod-29742453 is in running state.Got IP 10.47.255.250
2025-06-19 17:17:17,511 - DEBUG - Pod ctest-nginx-pod-29742453 has vmi 373218b8-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:17:17,511 - INFO - Verified pod ctest-nginx-pod-29742453 in contrail-api
2025-06-19 17:17:17,511 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:17:17,523 - DEBUG - VMI 373218b8-4d31-11f0-8532-fa163ede8ab3 is active in agent 10.0.0.27
2025-06-19 17:17:17,523 - INFO - Verified Pod ctest-nginx-pod-29742453 in agent 10.0.0.27
2025-06-19 17:17:17,523 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_VirtualMachineDatabaseList?x=6ce4b802-4157-47d1-b514-caeec194f694
2025-06-19 17:17:17,528 - INFO - Pod ctest-nginx-pod-29742453 with uuid 6ce4b802-4157-47d1-b514-caeec194f694 found in kube manager
2025-06-19 17:17:17,528 - INFO - Pod ctest-nginx-pod-29742453 verification passed
2025-06-19 17:17:17,580 - DEBUG - [Pod ctest-nginx-pod-29742453] Cmd: echo ctest-nginx-pod-29742453 > /usr/share/nginx/html/index.html, Output:
2025-06-19 17:17:17,586 - DEBUG - Pod ctest-busybox-pod-80609421 not in running state.Currently in Pending
2025-06-19 17:17:22,593 - INFO - Pod ctest-busybox-pod-80609421 is in running state.Got IP 10.47.255.248
2025-06-19 17:17:22,610 - DEBUG - Pod ctest-busybox-pod-80609421 has vmi 3a60455a-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:17:22,610 - INFO - Verified pod ctest-busybox-pod-80609421 in contrail-api
2025-06-19 17:17:22,611 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:17:22,623 - DEBUG - VMI 3a60455a-4d31-11f0-8532-fa163ede8ab3 is active in agent 10.0.0.27
2025-06-19 17:17:22,623 - INFO - Verified Pod ctest-busybox-pod-80609421 in agent 10.0.0.27
2025-06-19 17:17:22,623 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_VirtualMachineDatabaseList?x=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:22,629 - INFO - Pod ctest-busybox-pod-80609421 with uuid a84d9f69-17c3-492b-90ae-11d8e4371810 found in kube manager
2025-06-19 17:17:22,629 - INFO - Pod ctest-busybox-pod-80609421 verification passed
2025-06-19 17:17:22,682 - DEBUG - [Pod ctest-busybox-pod-80609421] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-29742453
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-06-19 17:17:22,682 - DEBUG - [Pod Pod ctest-busybox-pod-80609421] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-06-19 17:17:22,682 - DEBUG - [Pod Pod ctest-busybox-pod-80609421] Cmd output: ctest-nginx-pod-29742453
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-06-19 17:17:22,735 - DEBUG - [Pod ctest-busybox-pod-80609421] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-29742453
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-06-19 17:17:22,736 - DEBUG - [Pod Pod ctest-busybox-pod-80609421] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-06-19 17:17:22,736 - DEBUG - [Pod Pod ctest-busybox-pod-80609421] Cmd output: ctest-nginx-pod-29742453
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-06-19 17:17:22,792 - DEBUG - [Pod ctest-busybox-pod-80609421] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-30777511
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-06-19 17:17:22,793 - DEBUG - [Pod Pod ctest-busybox-pod-80609421] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-06-19 17:17:22,793 - DEBUG - [Pod Pod ctest-busybox-pod-80609421] Cmd output: ctest-nginx-pod-30777511
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-06-19 17:17:22,793 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-30777511': 1, 'ctest-nginx-pod-29742453': 2}
2025-06-19 17:17:22,793 - WARNING - Nothing to delete parallely
2025-06-19 17:17:22,793 - INFO - Deleting pod default:ctest-busybox-pod-80609421
2025-06-19 17:17:22,811 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:22,820 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:27,821 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:27,829 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:32,831 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:32,839 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:37,840 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:37,850 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:42,851 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:42,860 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:47,861 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:47,871 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:52,872 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:52,881 - WARNING - Pod uuid a84d9f69-17c3-492b-90ae-11d8e4371810 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:17:57,882 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a84d9f69-17c3-492b-90ae-11d8e4371810
2025-06-19 17:17:57,892 - DEBUG - Pod a84d9f69-17c3-492b-90ae-11d8e4371810 is not in agent 10.0.0.27 VM list
2025-06-19 17:17:57,892 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:3a60455a-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:17:57,899 - DEBUG - VMI 3a60455a-4d31-11f0-8532-fa163ede8ab3 is removed from agent 10.0.0.27
2025-06-19 17:17:57,899 - INFO - Verified that pod ctest-busybox-pod-80609421 is removed in agent
2025-06-19 17:17:57,899 - INFO - Deleting Ingress : ctest-nginx-ingress-36501067
2025-06-19 17:17:57,907 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=62205333-9487-4459-a986-5a79ffc3960c
2025-06-19 17:17:57,914 - ERROR - Ingress ctest-nginx-ingress-36501067 with uuid 62205333-9487-4459-a986-5a79ffc3960c still found in kube manager
2025-06-19 17:17:58,915 - DEBUG - Requesting: http://10.0.0.65:8108/Snh_LoadbalancerDatabaseList?x=62205333-9487-4459-a986-5a79ffc3960c
2025-06-19 17:17:58,919 - INFO - Ingress ctest-nginx-ingress-36501067 with uuid 62205333-9487-4459-a986-5a79ffc3960c deleted successfully from kube manager
2025-06-19 17:17:58,920 - INFO - Verifications on Ingress %s cleanup passed
2025-06-19 17:17:58,920 - INFO - Deleting the FIP pool __fip_pool_public__
2025-06-19 17:17:59,054 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-06-19 17:17:59,061 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-06-19 17:17:59,061 - INFO - Deleting VN __public__
2025-06-19 17:17:59,352 - DEBUG - Requesting: http://10.0.0.65:8082/routing-instance/2d9dd996-6643-4162-8ca3-792cef921049
2025-06-19 17:17:59,365 - DEBUG - Response Code: 404
2025-06-19 17:17:59,365 - DEBUG - Requesting: http://10.0.0.65:8082/domains
2025-06-19 17:17:59,377 - DEBUG - Requesting: http://10.0.0.65:8082/domain/6a247229-626e-4326-88a5-1fdd19f4d2d6
2025-06-19 17:17:59,396 - DEBUG - Requesting: http://10.0.0.65:8082/project/63c221f2-a419-4998-ab2d-4cbb7feb74f2
2025-06-19 17:17:59,491 - INFO - Validated that VN __public__ is not found in API Server
2025-06-19 17:17:59,491 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-06-19 17:17:59,503 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:59,514 - DEBUG - VN __public__ is not present in Agent 10.0.0.27
2025-06-19 17:17:59,514 - INFO - Validated that VN __public__ is not in any agent
2025-06-19 17:17:59,514 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:59,524 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-06-19 17:17:59,532 - DEBUG - Requesting: http://10.0.0.65:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-06-19 17:17:59,560 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-06-19 17:17:59,560 - INFO - Deleting pod default:ctest-nginx-pod-29742453
2025-06-19 17:17:59,584 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=6ce4b802-4157-47d1-b514-caeec194f694
2025-06-19 17:17:59,610 - WARNING - Pod uuid 6ce4b802-4157-47d1-b514-caeec194f694 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:18:04,611 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=6ce4b802-4157-47d1-b514-caeec194f694
2025-06-19 17:18:04,621 - DEBUG - Pod 6ce4b802-4157-47d1-b514-caeec194f694 is not in agent 10.0.0.27 VM list
2025-06-19 17:18:04,621 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:373218b8-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:18:04,627 - DEBUG - VMI 373218b8-4d31-11f0-8532-fa163ede8ab3 is removed from agent 10.0.0.27
2025-06-19 17:18:04,628 - INFO - Verified that pod ctest-nginx-pod-29742453 is removed in agent
2025-06-19 17:18:04,628 - INFO - Deleting pod default:ctest-nginx-pod-30777511
2025-06-19 17:18:04,644 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3ac818a1-e66f-488b-9094-2046e41b7179
2025-06-19 17:18:04,655 - WARNING - Pod uuid 3ac818a1-e66f-488b-9094-2046e41b7179 is still seen in agent 10.0.0.27 VM list
2025-06-19 17:18:09,655 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3ac818a1-e66f-488b-9094-2046e41b7179
2025-06-19 17:18:09,662 - DEBUG - Pod 3ac818a1-e66f-488b-9094-2046e41b7179 is not in agent 10.0.0.27 VM list
2025-06-19 17:18:09,662 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:36ae0d0c-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:18:09,670 - DEBUG - VMI 36ae0d0c-4d31-11f0-8532-fa163ede8ab3 is removed from agent 10.0.0.27
2025-06-19 17:18:09,670 - INFO - Verified that pod ctest-nginx-pod-30777511 is removed in agent
2025-06-19 17:18:09,670 - INFO - Deleting service : ctest-nginx-svc-98525832
2025-06-19 17:18:09,881 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AgentXmppConnectionStatusReq?
2025-06-19 17:18:09,892 - DEBUG - No XMPP flaps were noticed during the test
2025-06-19 17:18:09,892 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:11]
2025-06-19 17:18:09,892 - INFO - --------------------------------------------------------------------------------
2025-06-19 17:18:16,370 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-06-19 17:18:16,598 - DEBUG - Output : cluster_project={}
2025-06-19 17:18:16,719 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-06-19 17:18:16,904 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-06-19 17:18:16,910 - DEBUG - [10.0.0.65]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-06-19 17:18:17,102 - DEBUG - Output : cluster_project={}
2025-06-19 17:18:32,254 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:32,339 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:32,382 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:32,468 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:32,551 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:32,594 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:32,676 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:37,726 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-06-19 17:18:37,957 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AgentXmppConnectionStatusReq?
2025-06-19 17:18:40,891 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_ApplicationPolicySetReq?
2025-06-19 17:18:40,899 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AclReq?
2025-06-19 17:18:40,932 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AclReq?
2025-06-19 17:18:42,965 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AclReq?
2025-06-19 17:21:43,065 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_AgentXmppConnectionStatusReq?
2025-06-19 17:21:43,125 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a9c5a25a-1b06-42eb-a65a-1c8c1c2ca8c6
2025-06-19 17:21:43,135 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:21:43,141 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3b9f2d8b-2ceb-44fc-9d22-dc7481445268
2025-06-19 17:21:43,146 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=63130ec8-b92c-4afd-8d30-28742b890cd4
2025-06-19 17:21:43,157 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:21:43,163 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:21:43,169 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=f84c61de-a7f1-41ad-baa0-090b625489d4
2025-06-19 17:21:43,181 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:21:48,291 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=63130ec8-b92c-4afd-8d30-28742b890cd4
2025-06-19 17:21:48,293 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=a9c5a25a-1b06-42eb-a65a-1c8c1c2ca8c6
2025-06-19 17:21:48,295 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:21:48,296 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:21:48,297 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=f84c61de-a7f1-41ad-baa0-090b625489d4
2025-06-19 17:21:48,300 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:21:48,301 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3b9f2d8b-2ceb-44fc-9d22-dc7481445268
2025-06-19 17:21:48,302 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:21:48,306 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:64cdfe0e-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:21:48,332 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6448371a-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:21:48,334 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:658a3aba-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:21:48,336 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6613b0e2-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:21:53,329 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:21:53,333 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:21:53,334 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:21:53,336 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:21:58,340 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:21:58,349 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:21:58,350 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:21:58,352 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:22:03,348 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:22:03,363 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:22:03,365 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:22:03,367 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:22:08,356 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:22:08,378 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:22:08,380 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:22:08,383 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:22:13,365 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:22:13,393 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:22:13,395 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:22:13,396 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:22:18,390 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e727f97d-4550-46ac-9e7e-213245b35ede
2025-06-19 17:22:18,397 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6834d02c-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:22:18,408 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=e817870a-d850-4274-a039-60ddfb25e968
2025-06-19 17:22:18,411 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=38a2204d-a369-400e-a901-0f718feb2649
2025-06-19 17:22:18,412 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_VmListReq?uuid=3fa52588-811c-4ff8-9dac-05c29f0d9a24
2025-06-19 17:22:18,425 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:67308a72-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:22:18,427 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:66a83668-4d31-11f0-8532-fa163ede8ab3
2025-06-19 17:22:18,431 - DEBUG - Requesting: http://10.0.0.27:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:67b71bf0-4d31-11f0-8532-fa163ede8ab3