2025-09-15 16:00:58,123 - DEBUG - [10.0.0.50]: Running cmd : hostname
2025-09-15 16:00:58,375 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1481-1
2025-09-15 16:00:58,376 - DEBUG - [10.0.0.50]: Running cmd : hostname -f
2025-09-15 16:00:58,410 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1481-1
2025-09-15 16:00:58,410 - DEBUG - [10.0.0.50]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-09-15 16:00:58,475 - DEBUG - Output : NAMES
contrail_test_vtaqNBML9
contrail-kubernetes-master-kubemanager-1
webui-web-1
webui-job-1
analytics_database-nodemgr-1
analytics_database-cassandra-1
analytics_database-query-engine-1
analytics_snmp-nodemgr-1
analytics_snmp-topology-1
analytics_snmp-snmp-collector-1
analytics_alarm-alarm-gen-1
analytics_alarm-nodemgr-1
analytics_alarm-kafka-1
analytics-nodemgr-1
analytics-collector-1
analytics-api-1
redis-redis-1
control-named-1
control-nodemgr-1
control-dns-1
control-control-1
config_database-cassandra-1
config_database-zookeeper-1
config_database-rabbitmq-1
config_database-nodemgr-1
config_api-devicemgr-1
config_api-api-1
config_api-svcmonitor-1
config_api-dnsmasq-1
config_api-nodemgr-1
config_api-schema-1
2025-09-15 16:00:58,475 - DEBUG - [10.0.0.50]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-09-15 16:00:58,561 - DEBUG - Output : 127.0.0.1/8
10.0.0.50/24
252.50.0.1/8
10.124.232.1/24
172.17.0.1/16
2025-09-15 16:00:58,562 - DEBUG - [10.0.0.50]: Running cmd : getent hosts 10.0.0.50 | head -n 1 | awk '{print $2}'
2025-09-15 16:00:58,607 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1481-1
2025-09-15 16:00:58,734 - DEBUG - [10.0.0.33]: Running cmd : hostname
2025-09-15 16:00:58,991 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1481-1
2025-09-15 16:00:58,991 - DEBUG - [10.0.0.33]: Running cmd : hostname -f
2025-09-15 16:00:59,027 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1481-1
2025-09-15 16:00:59,028 - DEBUG - [10.0.0.33]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-09-15 16:00:59,108 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-09-15 16:00:59,108 - DEBUG - [10.0.0.33]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-09-15 16:00:59,142 - DEBUG - Output : 10.0.0.33/24
2025-09-15 16:00:59,142 - DEBUG - [10.0.0.33]: Running cmd : getent hosts 10.0.0.33 | head -n 1 | awk '{print $2}'
2025-09-15 16:00:59,178 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1481-1
2025-09-15 16:00:59,250 - INFO - Using existing project ['default-domain', 'k8s-default'](34d60028-c356-4605-88f4-8a077a90b872)
2025-09-15 16:00:59,374 - INFO - Using existing project ['default-domain', 'k8s-default'](34d60028-c356-4605-88f4-8a077a90b872)
2025-09-15 16:00:59,499 - INFO - ================================================================================
2025-09-15 16:00:59,499 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-09-15 16:00:59,500 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-09-15 16:00:59,668 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 16:00:59,679 - DEBUG - Nothing to compare xmpp stats {'10.0.0.33': {'10.0.0.50': '0'}} with
2025-09-15 16:00:59,679 - INFO - Initial checks done. Running the testcase now
2025-09-15 16:00:59,679 - INFO -
2025-09-15 16:00:59,684 - INFO - Namespace default already exists
2025-09-15 16:00:59,684 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-09-15 16:00:59,803 - DEBUG - Output : cluster_project={}
2025-09-15 16:00:59,865 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-09-15 16:00:59,866 - DEBUG - Project uuid in Contrail is 34d60028-c356-4605-88f4-8a077a90b872
2025-09-15 16:00:59,866 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-09-15 16:00:59,994 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-09-15 16:00:59,994 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_NamespaceDatabaseList?namespace_uuid=d60af2b3-c1ae-48ce-b6e5-0791df56f73e
2025-09-15 16:00:59,999 - INFO - Namespace default with uuid d60af2b3-c1ae-48ce-b6e5-0791df56f73e found in kube manager
2025-09-15 16:00:59,999 - INFO - Namespace default verification passed
2025-09-15 16:01:00,003 - DEBUG - Service ctest-nginx-svc-02647033 not present
2025-09-15 16:01:00,003 - INFO - Creating service ctest-nginx-svc-02647033
2025-09-15 16:01:00,025 - INFO - Created Service ctest-nginx-svc-02647033
2025-09-15 16:01:00,036 - DEBUG - Pod ctest-nginx-pod-98579703 not present
2025-09-15 16:01:00,036 - INFO - Creating Pod ctest-nginx-pod-98579703
2025-09-15 16:01:00,097 - DEBUG - Node for Pod ctest-nginx-pod-98579703 not yet populated
2025-09-15 16:01:03,106 - DEBUG - Pod : ctest-nginx-pod-98579703 UUID is 4155ab3f-0967-41f9-8d9c-b6aa904e3e9d
2025-09-15 16:01:03,116 - DEBUG - Pod ctest-nginx-pod-76079683 not present
2025-09-15 16:01:03,116 - INFO - Creating Pod ctest-nginx-pod-76079683
2025-09-15 16:01:03,143 - DEBUG - Pod : ctest-nginx-pod-76079683 UUID is f80333d5-fd65-4515-95a0-8bf44dd6158a
2025-09-15 16:01:03,391 - INFO - Created VN __public__, UUID :af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:03,806 - DEBUG - Requesting: http://10.0.0.50:8082/domains
2025-09-15 16:01:03,821 - DEBUG - Requesting: http://10.0.0.50:8082/domain/9b743d09-8a20-446a-8826-0a01d165dfb7
2025-09-15 16:01:03,841 - DEBUG - Requesting: http://10.0.0.50:8082/project/34d60028-c356-4605-88f4-8a077a90b872
2025-09-15 16:01:03,948 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:03,964 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:03,973 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/55aa1a3e-9816-4246-b2b3-849e3cb8f072
2025-09-15 16:01:03,980 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/55aa1a3e-9816-4246-b2b3-849e3cb8f072
2025-09-15 16:01:03,988 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/2ead6865-93e9-4bf1-93b2-cb9a64e474ac
2025-09-15 16:01:04,001 - DEBUG - Route Targets: ['target:64512:8000006']
2025-09-15 16:01:04,001 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:04,021 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/55aa1a3e-9816-4246-b2b3-849e3cb8f072
2025-09-15 16:01:04,054 - INFO - Verified VN network id 13 for VN af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:04,055 - INFO - Verifications in API Server for VN __public__ passed
2025-09-15 16:01:04,055 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:04,075 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/55aa1a3e-9816-4246-b2b3-849e3cb8f072
2025-09-15 16:01:04,097 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/55aa1a3e-9816-4246-b2b3-849e3cb8f072
2025-09-15 16:01:04,106 - DEBUG - Requesting: http://10.0.0.50:8082/route-target/2ead6865-93e9-4bf1-93b2-cb9a64e474ac
2025-09-15 16:01:04,113 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-09-15 16:01:04,123 - DEBUG - Control-node 10.0.0.50 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '12648944307232000113', 'uuid-lslong': '11688188190923632845'}, 'enable': 'true', 'created': '2025-09-15T16:01:03', 'last-modified': '2025-09-15T16:01:03', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.454799'}
2025-09-15 16:01:04,123 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:01:04,133 - DEBUG - Route Targets: ['target:64512:8000006']
2025-09-15 16:01:04,133 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-09-15 16:01:04,134 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-09-15 16:01:04,134 - DEBUG - Requesting: http://10.0.0.50:8082/domains
2025-09-15 16:01:04,150 - DEBUG - Requesting: http://10.0.0.50:8082/domain/9b743d09-8a20-446a-8826-0a01d165dfb7
2025-09-15 16:01:04,161 - DEBUG - Requesting: http://10.0.0.50:8082/project/34d60028-c356-4605-88f4-8a077a90b872
2025-09-15 16:01:04,222 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:04,233 - DEBUG - =>VN __public__ has no policy to be verified
2025-09-15 16:01:04,233 - DEBUG - Verifying the vn in opserver
2025-09-15 16:01:04,233 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.50
2025-09-15 16:01:04,234 - DEBUG - Requesting: http://10.0.0.50:8081/analytics/uves/virtual-networks
2025-09-15 16:01:04,239 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.50:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-09-15 16:01:04,239 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-09-15 16:01:04,239 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:01:04,248 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:01:04,255 - DEBUG - Do not have enough data to verify VN in agent
2025-09-15 16:01:04,255 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:01:04,264 - DEBUG - VRF ids for VN __public__: {}
2025-09-15 16:01:04,412 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-09-15 16:01:04,531 - DEBUG - Requesting: http://10.0.0.50:8082/domains
2025-09-15 16:01:04,544 - DEBUG - Requesting: http://10.0.0.50:8082/domain/9b743d09-8a20-446a-8826-0a01d165dfb7
2025-09-15 16:01:04,562 - DEBUG - Requesting: http://10.0.0.50:8082/project/34d60028-c356-4605-88f4-8a077a90b872
2025-09-15 16:01:04,607 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:04,625 - DEBUG - Requesting: http://10.0.0.50:8082/floating-ip-pool/49a050a2-2dd1-4067-b5f5-c196a7680b23
2025-09-15 16:01:04,632 - DEBUG - Requesting: http://10.0.0.50:8082/domains
2025-09-15 16:01:04,638 - DEBUG - Requesting: http://10.0.0.50:8082/domain/9b743d09-8a20-446a-8826-0a01d165dfb7
2025-09-15 16:01:04,647 - DEBUG - Requesting: http://10.0.0.50:8082/project/34d60028-c356-4605-88f4-8a077a90b872
2025-09-15 16:01:04,698 - DEBUG - Requesting: http://10.0.0.50:8082/virtual-network/af8a13d4-9843-4871-a234-c93ba3eb30cd
2025-09-15 16:01:04,710 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-09-15 16:01:04,710 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-09-15 16:01:04,719 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-09-15 16:01:04,724 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-09-15 16:01:04,724 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-09-15 16:01:04,727 - DEBUG - Ingress ctest-nginx-ingress-14052665 not present
2025-09-15 16:01:04,728 - INFO - Creating Ingress ctest-nginx-ingress-14052665
2025-09-15 16:01:04,736 - INFO - Created Ingress ctest-nginx-ingress-14052665
2025-09-15 16:01:04,740 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-14052665
2025-09-15 16:01:07,749 - DEBUG - For Ingress ctest-nginx-ingress-14052665, Cluster IP: 10.47.255.249, External IPs ['150.113.81.124']
2025-09-15 16:01:07,749 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-14052665
2025-09-15 16:01:07,750 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=224a6b10-71bc-4990-9ecd-b41618804e4b
2025-09-15 16:01:07,755 - INFO - Ingress ctest-nginx-ingress-14052665 with uuid 224a6b10-71bc-4990-9ecd-b41618804e4b found in kube manager
2025-09-15 16:01:07,755 - INFO - Ingress ctest-nginx-ingress-14052665 verification passed
2025-09-15 16:01:07,766 - DEBUG - Pod ctest-busybox-pod-43803288 not present
2025-09-15 16:01:07,766 - INFO - Creating Pod ctest-busybox-pod-43803288
2025-09-15 16:01:07,808 - DEBUG - Pod : ctest-busybox-pod-43803288 UUID is 9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:07,826 - DEBUG - Pod ctest-nginx-pod-98579703 not in running state.Currently in Pending
2025-09-15 16:01:12,837 - DEBUG - Pod ctest-nginx-pod-98579703 not in running state.Currently in Pending
2025-09-15 16:01:17,846 - INFO - Pod ctest-nginx-pod-98579703 is in running state.Got IP 10.47.255.251
2025-09-15 16:01:17,861 - DEBUG - Pod ctest-nginx-pod-98579703 has vmi 2d372202-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:01:17,861 - INFO - Verified pod ctest-nginx-pod-98579703 in contrail-api
2025-09-15 16:01:17,862 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:01:17,873 - DEBUG - VMI 2d372202-924d-11f0-9d9f-fa163e37ffae is active in agent 10.0.0.33
2025-09-15 16:01:17,873 - INFO - Verified Pod ctest-nginx-pod-98579703 in agent 10.0.0.33
2025-09-15 16:01:17,874 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=4155ab3f-0967-41f9-8d9c-b6aa904e3e9d
2025-09-15 16:01:17,878 - INFO - Pod ctest-nginx-pod-98579703 with uuid 4155ab3f-0967-41f9-8d9c-b6aa904e3e9d found in kube manager
2025-09-15 16:01:17,878 - INFO - Pod ctest-nginx-pod-98579703 verification passed
2025-09-15 16:01:17,956 - DEBUG - [Pod ctest-nginx-pod-98579703] Cmd: echo ctest-nginx-pod-98579703 > /usr/share/nginx/html/index.html, Output:
2025-09-15 16:01:17,965 - INFO - Pod ctest-nginx-pod-76079683 is in running state.Got IP 10.47.255.250
2025-09-15 16:01:18,024 - DEBUG - Pod ctest-nginx-pod-76079683 has vmi 2e2b34c8-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:01:18,024 - INFO - Verified pod ctest-nginx-pod-76079683 in contrail-api
2025-09-15 16:01:18,025 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:01:18,038 - DEBUG - VMI 2e2b34c8-924d-11f0-9d9f-fa163e37ffae is active in agent 10.0.0.33
2025-09-15 16:01:18,039 - INFO - Verified Pod ctest-nginx-pod-76079683 in agent 10.0.0.33
2025-09-15 16:01:18,039 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=f80333d5-fd65-4515-95a0-8bf44dd6158a
2025-09-15 16:01:18,044 - INFO - Pod ctest-nginx-pod-76079683 with uuid f80333d5-fd65-4515-95a0-8bf44dd6158a found in kube manager
2025-09-15 16:01:18,044 - INFO - Pod ctest-nginx-pod-76079683 verification passed
2025-09-15 16:01:18,098 - DEBUG - [Pod ctest-nginx-pod-76079683] Cmd: echo ctest-nginx-pod-76079683 > /usr/share/nginx/html/index.html, Output:
2025-09-15 16:01:18,106 - DEBUG - Pod ctest-busybox-pod-43803288 not in running state.Currently in Pending
2025-09-15 16:01:23,114 - INFO - Pod ctest-busybox-pod-43803288 is in running state.Got IP 10.47.255.248
2025-09-15 16:01:23,128 - DEBUG - Pod ctest-busybox-pod-43803288 has vmi 31242040-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:01:23,128 - INFO - Verified pod ctest-busybox-pod-43803288 in contrail-api
2025-09-15 16:01:23,128 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:01:23,142 - DEBUG - VMI 31242040-924d-11f0-9d9f-fa163e37ffae is active in agent 10.0.0.33
2025-09-15 16:01:23,142 - INFO - Verified Pod ctest-busybox-pod-43803288 in agent 10.0.0.33
2025-09-15 16:01:23,143 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_VirtualMachineDatabaseList?x=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:23,147 - INFO - Pod ctest-busybox-pod-43803288 with uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e found in kube manager
2025-09-15 16:01:23,148 - INFO - Pod ctest-busybox-pod-43803288 verification passed
2025-09-15 16:01:23,216 - DEBUG - [Pod ctest-busybox-pod-43803288] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-98579703
2025-09-15 16:01:23,216 - DEBUG - [Pod Pod ctest-busybox-pod-43803288] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-09-15 16:01:23,216 - DEBUG - [Pod Pod ctest-busybox-pod-43803288] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-98579703
2025-09-15 16:01:23,279 - DEBUG - [Pod ctest-busybox-pod-43803288] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-76079683
2025-09-15 16:01:23,279 - DEBUG - [Pod Pod ctest-busybox-pod-43803288] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-09-15 16:01:23,279 - DEBUG - [Pod Pod ctest-busybox-pod-43803288] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-76079683
2025-09-15 16:01:23,279 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-98579703': 1, 'ctest-nginx-pod-76079683': 1}
2025-09-15 16:01:23,279 - WARNING - Nothing to delete parallely
2025-09-15 16:01:23,279 - INFO - Deleting pod default:ctest-busybox-pod-43803288
2025-09-15 16:01:23,306 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:23,318 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:28,319 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:28,326 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:33,327 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:33,334 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:38,335 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:38,342 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:43,343 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:43,351 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:48,352 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:48,362 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:53,363 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:53,369 - WARNING - Pod uuid 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is still seen in agent 10.0.0.33 VM list
2025-09-15 16:01:58,370 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=9a85e0af-c66a-44f2-99c3-763f0fdbc77e
2025-09-15 16:01:58,378 - DEBUG - Pod 9a85e0af-c66a-44f2-99c3-763f0fdbc77e is not in agent 10.0.0.33 VM list
2025-09-15 16:01:58,378 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:31242040-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:01:58,385 - DEBUG - VMI 31242040-924d-11f0-9d9f-fa163e37ffae is removed from agent 10.0.0.33
2025-09-15 16:01:58,385 - INFO - Verified that pod ctest-busybox-pod-43803288 is removed in agent
2025-09-15 16:01:58,385 - INFO - Deleting Ingress : ctest-nginx-ingress-14052665
2025-09-15 16:01:58,395 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=224a6b10-71bc-4990-9ecd-b41618804e4b
2025-09-15 16:01:58,403 - ERROR - Ingress ctest-nginx-ingress-14052665 with uuid 224a6b10-71bc-4990-9ecd-b41618804e4b still found in kube manager
2025-09-15 16:01:59,403 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=224a6b10-71bc-4990-9ecd-b41618804e4b
2025-09-15 16:01:59,411 - ERROR - Ingress ctest-nginx-ingress-14052665 with uuid 224a6b10-71bc-4990-9ecd-b41618804e4b still found in kube manager
2025-09-15 16:02:00,411 - DEBUG - Requesting: http://10.0.0.50:8108/Snh_LoadbalancerDatabaseList?x=224a6b10-71bc-4990-9ecd-b41618804e4b
2025-09-15 16:02:00,417 - INFO - Ingress ctest-nginx-ingress-14052665 with uuid 224a6b10-71bc-4990-9ecd-b41618804e4b deleted successfully from kube manager
2025-09-15 16:02:00,418 - INFO - Verifications on Ingress %s cleanup passed
2025-09-15 16:02:00,418 - INFO - Deleting the FIP pool __fip_pool_public__
2025-09-15 16:02:00,546 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-09-15 16:02:00,562 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-09-15 16:02:00,563 - INFO - Deleting VN __public__
2025-09-15 16:02:00,888 - DEBUG - Requesting: http://10.0.0.50:8082/routing-instance/55aa1a3e-9816-4246-b2b3-849e3cb8f072
2025-09-15 16:02:00,935 - DEBUG - Response Code: 404
2025-09-15 16:02:00,935 - DEBUG - Requesting: http://10.0.0.50:8082/domains
2025-09-15 16:02:00,962 - DEBUG - Requesting: http://10.0.0.50:8082/domain/9b743d09-8a20-446a-8826-0a01d165dfb7
2025-09-15 16:02:00,990 - DEBUG - Requesting: http://10.0.0.50:8082/project/34d60028-c356-4605-88f4-8a077a90b872
2025-09-15 16:02:01,139 - INFO - Validated that VN __public__ is not found in API Server
2025-09-15 16:02:01,139 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-09-15 16:02:01,173 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:02:01,191 - DEBUG - VN __public__ is not present in Agent 10.0.0.33
2025-09-15 16:02:01,191 - INFO - Validated that VN __public__ is not in any agent
2025-09-15 16:02:01,192 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:02:01,205 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-09-15 16:02:01,221 - DEBUG - Requesting: http://10.0.0.50:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-09-15 16:02:01,249 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-09-15 16:02:01,249 - INFO - Deleting pod default:ctest-nginx-pod-76079683
2025-09-15 16:02:01,283 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=f80333d5-fd65-4515-95a0-8bf44dd6158a
2025-09-15 16:02:01,293 - WARNING - Pod uuid f80333d5-fd65-4515-95a0-8bf44dd6158a is still seen in agent 10.0.0.33 VM list
2025-09-15 16:02:06,293 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=f80333d5-fd65-4515-95a0-8bf44dd6158a
2025-09-15 16:02:06,301 - DEBUG - Pod f80333d5-fd65-4515-95a0-8bf44dd6158a is not in agent 10.0.0.33 VM list
2025-09-15 16:02:06,301 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2e2b34c8-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:02:06,308 - DEBUG - VMI 2e2b34c8-924d-11f0-9d9f-fa163e37ffae is removed from agent 10.0.0.33
2025-09-15 16:02:06,308 - INFO - Verified that pod ctest-nginx-pod-76079683 is removed in agent
2025-09-15 16:02:06,309 - INFO - Deleting pod default:ctest-nginx-pod-98579703
2025-09-15 16:02:06,339 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=4155ab3f-0967-41f9-8d9c-b6aa904e3e9d
2025-09-15 16:02:06,349 - WARNING - Pod uuid 4155ab3f-0967-41f9-8d9c-b6aa904e3e9d is still seen in agent 10.0.0.33 VM list
2025-09-15 16:02:11,349 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=4155ab3f-0967-41f9-8d9c-b6aa904e3e9d
2025-09-15 16:02:11,359 - DEBUG - Pod 4155ab3f-0967-41f9-8d9c-b6aa904e3e9d is not in agent 10.0.0.33 VM list
2025-09-15 16:02:11,359 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:2d372202-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:02:11,369 - DEBUG - VMI 2d372202-924d-11f0-9d9f-fa163e37ffae is removed from agent 10.0.0.33
2025-09-15 16:02:11,369 - INFO - Verified that pod ctest-nginx-pod-98579703 is removed in agent
2025-09-15 16:02:11,369 - INFO - Deleting service : ctest-nginx-svc-02647033
2025-09-15 16:02:11,655 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 16:02:11,667 - DEBUG - No XMPP flaps were noticed during the test
2025-09-15 16:02:11,667 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:12]
2025-09-15 16:02:11,667 - INFO - --------------------------------------------------------------------------------
2025-09-15 16:02:18,294 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-09-15 16:02:18,480 - DEBUG - Output : cluster_project={}
2025-09-15 16:02:18,609 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-09-15 16:02:18,804 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-09-15 16:02:18,810 - DEBUG - [10.0.0.50]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-09-15 16:02:18,985 - DEBUG - Output : cluster_project={}
2025-09-15 16:02:29,147 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:29,240 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:29,276 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:34,372 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:34,410 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:34,493 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:34,572 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:39,620 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-09-15 16:02:39,849 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 16:02:42,511 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_ApplicationPolicySetReq?
2025-09-15 16:02:42,518 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AclReq?
2025-09-15 16:02:42,544 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AclReq?
2025-09-15 16:02:44,571 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AclReq?
2025-09-15 16:05:45,112 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_AgentXmppConnectionStatusReq?
2025-09-15 16:05:45,179 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ecb8cc7d-244d-45eb-9d61-c86bad48429b
2025-09-15 16:05:45,195 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:05:45,201 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:05:45,208 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=4e620e0a-0c51-4bd8-94a4-f63aaebc6049
2025-09-15 16:05:45,212 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3018ed62-1907-4185-a168-b599947cb790
2025-09-15 16:05:45,227 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:05:45,241 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:05:45,247 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=e291752f-708b-468a-a4d3-2fa42fbbec01
2025-09-15 16:05:50,254 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=ecb8cc7d-244d-45eb-9d61-c86bad48429b
2025-09-15 16:05:50,261 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5bde65c0-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:05:50,270 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:05:50,271 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:05:50,313 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=e291752f-708b-468a-a4d3-2fa42fbbec01
2025-09-15 16:05:50,314 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:05:50,315 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=4e620e0a-0c51-4bd8-94a4-f63aaebc6049
2025-09-15 16:05:50,318 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3018ed62-1907-4185-a168-b599947cb790
2025-09-15 16:05:50,319 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:05:50,334 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5d4dfc54-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:05:50,336 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5c8aac36-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:05:50,339 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5dcdb0ca-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:05:55,282 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:05:55,285 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:05:55,336 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:05:55,341 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:06:00,294 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:06:00,296 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:06:00,345 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:06:00,349 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:06:05,306 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:06:05,308 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:06:05,354 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:06:05,359 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:06:10,318 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:06:10,320 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:06:10,362 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:06:10,367 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:06:15,334 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:06:15,336 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:06:15,372 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:06:15,377 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:06:20,347 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=3a16bf03-8d8e-4ddf-b3b4-e24259a4aefb
2025-09-15 16:06:20,349 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=74888c24-c756-47b7-bdf9-0c8ccb810851
2025-09-15 16:06:20,358 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5e651d0c-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:06:20,360 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5ef0c88e-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:06:20,380 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=08ec0cc4-bf92-4535-9753-f5d5b091f138
2025-09-15 16:06:20,386 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_VmListReq?uuid=6d16416a-feca-4874-8133-a6fcb9d1a4cf
2025-09-15 16:06:20,390 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:5fae1a88-924d-11f0-9d9f-fa163e37ffae
2025-09-15 16:06:20,401 - DEBUG - Requesting: http://10.0.0.33:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:60401406-924d-11f0-9d9f-fa163e37ffae