2025-11-12 08:40:29,117 - DEBUG - [10.0.0.22]: Running cmd : hostname
2025-11-12 08:40:29,365 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1722-1
2025-11-12 08:40:29,366 - DEBUG - [10.0.0.22]: Running cmd : hostname -f
2025-11-12 08:40:29,399 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1722-1
2025-11-12 08:40:29,400 - DEBUG - [10.0.0.22]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-11-12 08:40:29,456 - DEBUG - Output : NAMES
contrail_test_NyxFv6t4k
analytics_database-query-engine-1
analytics_database-cassandra-1
analytics_database-nodemgr-1
analytics_snmp-snmp-collector-1
analytics_snmp-nodemgr-1
analytics_snmp-topology-1
analytics_alarm-nodemgr-1
analytics_alarm-alarm-gen-1
analytics_alarm-kafka-1
analytics-collector-1
analytics-api-1
analytics-nodemgr-1
contrail-kubernetes-master-kubemanager-1
redis-redis-1
webui-web-1
webui-job-1
control-named-1
control-control-1
control-dns-1
control-nodemgr-1
config_database-zookeeper-1
config_database-rabbitmq-1
config_database-cassandra-1
config_database-nodemgr-1
config_api-devicemgr-1
config_api-schema-1
config_api-dnsmasq-1
config_api-api-1
config_api-nodemgr-1
config_api-svcmonitor-1
2025-11-12 08:40:29,456 - DEBUG - [10.0.0.22]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-11-12 08:40:29,544 - DEBUG - Output : 127.0.0.1/8
10.0.0.22/24
252.22.0.1/8
10.39.77.1/24
172.17.0.1/16
2025-11-12 08:40:29,544 - DEBUG - [10.0.0.22]: Running cmd : getent hosts 10.0.0.22 | head -n 1 | awk '{print $2}'
2025-11-12 08:40:29,579 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1722-1
2025-11-12 08:40:29,677 - DEBUG - [10.0.0.65]: Running cmd : hostname
2025-11-12 08:40:29,906 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1722-1
2025-11-12 08:40:29,906 - DEBUG - [10.0.0.65]: Running cmd : hostname -f
2025-11-12 08:40:29,940 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1722-1
2025-11-12 08:40:29,940 - DEBUG - [10.0.0.65]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-11-12 08:40:30,034 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-11-12 08:40:30,034 - DEBUG - [10.0.0.65]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-11-12 08:40:30,069 - DEBUG - Output : 10.0.0.65/24
2025-11-12 08:40:30,069 - DEBUG - [10.0.0.65]: Running cmd : getent hosts 10.0.0.65 | head -n 1 | awk '{print $2}'
2025-11-12 08:40:30,104 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1722-1
2025-11-12 08:40:30,171 - INFO - Using existing project ['default-domain', 'k8s-default'](a044a930-430b-4898-8313-96cb56adc7b9)
2025-11-12 08:40:30,282 - INFO - Using existing project ['default-domain', 'k8s-default'](a044a930-430b-4898-8313-96cb56adc7b9)
2025-11-12 08:40:30,409 - INFO - ================================================================================
2025-11-12 08:40:30,409 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-11-12 08:40:30,410 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-11-12 08:40:30,619 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq?
2025-11-12 08:40:30,625 - DEBUG - Nothing to compare xmpp stats {'10.0.0.65': {'10.0.0.22': '0'}} with
2025-11-12 08:40:30,625 - INFO - Initial checks done. Running the testcase now
2025-11-12 08:40:30,625 - INFO -
2025-11-12 08:40:30,630 - INFO - Namespace default already exists
2025-11-12 08:40:30,630 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-11-12 08:40:30,746 - DEBUG - Output : cluster_project={}
2025-11-12 08:40:30,821 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-11-12 08:40:30,823 - DEBUG - Project uuid in Contrail is a044a930-430b-4898-8313-96cb56adc7b9
2025-11-12 08:40:30,823 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-11-12 08:40:30,963 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-11-12 08:40:30,963 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_NamespaceDatabaseList?namespace_uuid=92a01a49-7f2c-4690-a461-7d98886b394f
2025-11-12 08:40:30,967 - INFO - Namespace default with uuid 92a01a49-7f2c-4690-a461-7d98886b394f found in kube manager
2025-11-12 08:40:30,968 - INFO - Namespace default verification passed
2025-11-12 08:40:30,971 - DEBUG - Service ctest-nginx-svc-03825539 not present
2025-11-12 08:40:30,972 - INFO - Creating service ctest-nginx-svc-03825539
2025-11-12 08:40:31,002 - INFO - Created Service ctest-nginx-svc-03825539
2025-11-12 08:40:31,012 - DEBUG - Pod ctest-nginx-pod-35328108 not present
2025-11-12 08:40:31,016 - INFO - Creating Pod ctest-nginx-pod-35328108
2025-11-12 08:40:31,048 - DEBUG - Node for Pod ctest-nginx-pod-35328108 not yet populated
2025-11-12 08:40:34,056 - DEBUG - Pod : ctest-nginx-pod-35328108 UUID is 6478989f-40c3-4b98-9d92-b9a9d292ace4
2025-11-12 08:40:34,058 - DEBUG - Pod ctest-nginx-pod-15544842 not present
2025-11-12 08:40:34,059 - INFO - Creating Pod ctest-nginx-pod-15544842
2025-11-12 08:40:34,096 - DEBUG - Pod : ctest-nginx-pod-15544842 UUID is c3b19062-5179-4729-99d6-75c23251d602
2025-11-12 08:40:34,323 - INFO - Created VN __public__, UUID :51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,588 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-11-12 08:40:34,594 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6c3209c3-6307-4767-8fd9-1e60a49bc523
2025-11-12 08:40:34,617 - DEBUG - Requesting: http://10.0.0.22:8082/project/a044a930-430b-4898-8313-96cb56adc7b9
2025-11-12 08:40:34,694 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,713 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,725 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/498ba4e6-4f18-40bb-a7f7-b001eede089f
2025-11-12 08:40:34,731 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/498ba4e6-4f18-40bb-a7f7-b001eede089f
2025-11-12 08:40:34,738 - DEBUG - Requesting: http://10.0.0.22:8082/route-target/c35fdcc6-2baf-4072-b7e4-17b335684bcd
2025-11-12 08:40:34,744 - DEBUG - Route Targets: ['target:64512:8000006']
2025-11-12 08:40:34,744 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,751 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/498ba4e6-4f18-40bb-a7f7-b001eede089f
2025-11-12 08:40:34,768 - INFO - Verified VN network id 13 for VN 51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,768 - INFO - Verifications in API Server for VN __public__ passed
2025-11-12 08:40:34,768 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,776 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/498ba4e6-4f18-40bb-a7f7-b001eede089f
2025-11-12 08:40:34,782 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/498ba4e6-4f18-40bb-a7f7-b001eede089f
2025-11-12 08:40:34,787 - DEBUG - Requesting: http://10.0.0.22:8082/route-target/c35fdcc6-2baf-4072-b7e4-17b335684bcd
2025-11-12 08:40:34,806 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-11-12 08:40:34,814 - DEBUG - Control-node 10.0.0.22 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '5898846599020498018', 'uuid-lslong': '13049371395155467817'}, 'enable': 'true', 'created': '2025-11-12T08:40:34', 'last-modified': '2025-11-12T08:40:34', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.304552'}
2025-11-12 08:40:34,814 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:40:34,825 - DEBUG - Route Targets: ['target:64512:8000006']
2025-11-12 08:40:34,825 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-11-12 08:40:34,826 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-11-12 08:40:34,826 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-11-12 08:40:34,831 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6c3209c3-6307-4767-8fd9-1e60a49bc523
2025-11-12 08:40:34,837 - DEBUG - Requesting: http://10.0.0.22:8082/project/a044a930-430b-4898-8313-96cb56adc7b9
2025-11-12 08:40:34,892 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:34,901 - DEBUG - =>VN __public__ has no policy to be verified
2025-11-12 08:40:34,901 - DEBUG - Verifying the vn in opserver
2025-11-12 08:40:34,901 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.22
2025-11-12 08:40:34,901 - DEBUG - Requesting: http://10.0.0.22:8081/analytics/uves/virtual-networks
2025-11-12 08:40:34,905 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.22:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-11-12 08:40:34,905 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-11-12 08:40:34,906 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:40:34,912 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:40:34,917 - DEBUG - Do not have enough data to verify VN in agent
2025-11-12 08:40:34,917 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:40:34,922 - DEBUG - VRF ids for VN __public__: {}
2025-11-12 08:40:35,028 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-11-12 08:40:35,135 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-11-12 08:40:35,140 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6c3209c3-6307-4767-8fd9-1e60a49bc523
2025-11-12 08:40:35,152 - DEBUG - Requesting: http://10.0.0.22:8082/project/a044a930-430b-4898-8313-96cb56adc7b9
2025-11-12 08:40:35,226 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:35,236 - DEBUG - Requesting: http://10.0.0.22:8082/floating-ip-pool/9d164a97-4fb3-4983-8ad2-c1dcdb121ee8
2025-11-12 08:40:35,242 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-11-12 08:40:35,246 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6c3209c3-6307-4767-8fd9-1e60a49bc523
2025-11-12 08:40:35,253 - DEBUG - Requesting: http://10.0.0.22:8082/project/a044a930-430b-4898-8313-96cb56adc7b9
2025-11-12 08:40:35,298 - DEBUG - Requesting: http://10.0.0.22:8082/virtual-network/51dce9ba-7723-4c62-b518-ae2536629e29
2025-11-12 08:40:35,311 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-11-12 08:40:35,311 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-11-12 08:40:35,319 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-11-12 08:40:35,324 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-11-12 08:40:35,324 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-11-12 08:40:35,327 - DEBUG - Ingress ctest-nginx-ingress-60830441 not present
2025-11-12 08:40:35,328 - INFO - Creating Ingress ctest-nginx-ingress-60830441
2025-11-12 08:40:35,336 - INFO - Created Ingress ctest-nginx-ingress-60830441
2025-11-12 08:40:35,340 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-60830441
2025-11-12 08:40:38,344 - DEBUG - For Ingress ctest-nginx-ingress-60830441, Cluster IP: 10.47.255.249, External IPs ['202.168.93.60']
2025-11-12 08:40:38,344 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-60830441
2025-11-12 08:40:38,344 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=b01a3905-83d0-422d-9b09-de93a91114f8
2025-11-12 08:40:38,348 - INFO - Ingress ctest-nginx-ingress-60830441 with uuid b01a3905-83d0-422d-9b09-de93a91114f8 found in kube manager
2025-11-12 08:40:38,349 - INFO - Ingress ctest-nginx-ingress-60830441 verification passed
2025-11-12 08:40:38,351 - DEBUG - Pod ctest-busybox-pod-79796430 not present
2025-11-12 08:40:38,351 - INFO - Creating Pod ctest-busybox-pod-79796430
2025-11-12 08:40:38,372 - DEBUG - Pod : ctest-busybox-pod-79796430 UUID is a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:40:38,389 - DEBUG - Pod ctest-nginx-pod-35328108 not in running state.Currently in Pending
2025-11-12 08:40:43,411 - INFO - Pod ctest-nginx-pod-35328108 is in running state.Got IP 10.47.255.251
2025-11-12 08:40:43,484 - DEBUG - Pod ctest-nginx-pod-35328108 has vmi 402e9114-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:40:43,484 - INFO - Verified pod ctest-nginx-pod-35328108 in contrail-api
2025-11-12 08:40:43,484 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:40:43,498 - DEBUG - VMI 402e9114-bfa3-11f0-bc61-fa163ea22c9e is active in agent 10.0.0.65
2025-11-12 08:40:43,498 - INFO - Verified Pod ctest-nginx-pod-35328108 in agent 10.0.0.65
2025-11-12 08:40:43,498 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=6478989f-40c3-4b98-9d92-b9a9d292ace4
2025-11-12 08:40:43,511 - INFO - Pod ctest-nginx-pod-35328108 with uuid 6478989f-40c3-4b98-9d92-b9a9d292ace4 found in kube manager
2025-11-12 08:40:43,512 - INFO - Pod ctest-nginx-pod-35328108 verification passed
2025-11-12 08:40:43,596 - DEBUG - [Pod ctest-nginx-pod-35328108] Cmd: echo ctest-nginx-pod-35328108 > /usr/share/nginx/html/index.html, Output:
2025-11-12 08:40:43,605 - DEBUG - Pod ctest-nginx-pod-15544842 not in running state.Currently in Pending
2025-11-12 08:40:48,613 - INFO - Pod ctest-nginx-pod-15544842 is in running state.Got IP 10.47.255.250
2025-11-12 08:40:48,627 - DEBUG - Pod ctest-nginx-pod-15544842 has vmi 412f37ee-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:40:48,628 - INFO - Verified pod ctest-nginx-pod-15544842 in contrail-api
2025-11-12 08:40:48,628 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:40:48,639 - DEBUG - VMI 412f37ee-bfa3-11f0-bc61-fa163ea22c9e is active in agent 10.0.0.65
2025-11-12 08:40:48,639 - INFO - Verified Pod ctest-nginx-pod-15544842 in agent 10.0.0.65
2025-11-12 08:40:48,639 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=c3b19062-5179-4729-99d6-75c23251d602
2025-11-12 08:40:48,644 - INFO - Pod ctest-nginx-pod-15544842 with uuid c3b19062-5179-4729-99d6-75c23251d602 found in kube manager
2025-11-12 08:40:48,644 - INFO - Pod ctest-nginx-pod-15544842 verification passed
2025-11-12 08:40:48,707 - DEBUG - [Pod ctest-nginx-pod-15544842] Cmd: echo ctest-nginx-pod-15544842 > /usr/share/nginx/html/index.html, Output:
2025-11-12 08:40:48,713 - DEBUG - Pod ctest-busybox-pod-79796430 not in running state.Currently in Pending
2025-11-12 08:40:53,734 - INFO - Pod ctest-busybox-pod-79796430 is in running state.Got IP 10.47.255.248
2025-11-12 08:40:53,747 - DEBUG - Pod ctest-busybox-pod-79796430 has vmi 43c23f10-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:40:53,747 - INFO - Verified pod ctest-busybox-pod-79796430 in contrail-api
2025-11-12 08:40:53,747 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:40:53,756 - DEBUG - VMI 43c23f10-bfa3-11f0-bc61-fa163ea22c9e is active in agent 10.0.0.65
2025-11-12 08:40:53,756 - INFO - Verified Pod ctest-busybox-pod-79796430 in agent 10.0.0.65
2025-11-12 08:40:53,756 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_VirtualMachineDatabaseList?x=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:40:53,760 - INFO - Pod ctest-busybox-pod-79796430 with uuid a4ca68a3-6029-4b8c-bf3d-782368137acb found in kube manager
2025-11-12 08:40:53,760 - INFO - Pod ctest-busybox-pod-79796430 verification passed
2025-11-12 08:40:53,810 - DEBUG - [Pod ctest-busybox-pod-79796430] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-15544842
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-11-12 08:40:53,810 - DEBUG - [Pod Pod ctest-busybox-pod-79796430] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-11-12 08:40:53,810 - DEBUG - [Pod Pod ctest-busybox-pod-79796430] Cmd output: ctest-nginx-pod-15544842
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-11-12 08:40:53,858 - DEBUG - [Pod ctest-busybox-pod-79796430] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-35328108
2025-11-12 08:40:53,858 - DEBUG - [Pod Pod ctest-busybox-pod-79796430] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-11-12 08:40:53,858 - DEBUG - [Pod Pod ctest-busybox-pod-79796430] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-35328108
2025-11-12 08:40:53,858 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-35328108': 1, 'ctest-nginx-pod-15544842': 1}
2025-11-12 08:40:53,858 - WARNING - Nothing to delete parallely
2025-11-12 08:40:53,859 - INFO - Deleting pod default:ctest-busybox-pod-79796430
2025-11-12 08:40:53,872 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:40:53,878 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:40:58,879 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:40:58,886 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:03,886 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:41:03,892 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:08,893 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:41:08,900 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:13,902 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:41:13,912 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:18,914 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:41:18,922 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:23,923 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:41:23,929 - WARNING - Pod uuid a4ca68a3-6029-4b8c-bf3d-782368137acb is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:28,930 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=a4ca68a3-6029-4b8c-bf3d-782368137acb
2025-11-12 08:41:28,936 - DEBUG - Pod a4ca68a3-6029-4b8c-bf3d-782368137acb is not in agent 10.0.0.65 VM list
2025-11-12 08:41:28,936 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:43c23f10-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:41:28,941 - DEBUG - VMI 43c23f10-bfa3-11f0-bc61-fa163ea22c9e is removed from agent 10.0.0.65
2025-11-12 08:41:28,941 - INFO - Verified that pod ctest-busybox-pod-79796430 is removed in agent
2025-11-12 08:41:28,941 - INFO - Deleting Ingress : ctest-nginx-ingress-60830441
2025-11-12 08:41:28,951 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=b01a3905-83d0-422d-9b09-de93a91114f8
2025-11-12 08:41:28,957 - ERROR - Ingress ctest-nginx-ingress-60830441 with uuid b01a3905-83d0-422d-9b09-de93a91114f8 still found in kube manager
2025-11-12 08:41:29,958 - DEBUG - Requesting: http://10.0.0.22:8108/Snh_LoadbalancerDatabaseList?x=b01a3905-83d0-422d-9b09-de93a91114f8
2025-11-12 08:41:29,962 - INFO - Ingress ctest-nginx-ingress-60830441 with uuid b01a3905-83d0-422d-9b09-de93a91114f8 deleted successfully from kube manager
2025-11-12 08:41:29,962 - INFO - Verifications on Ingress %s cleanup passed
2025-11-12 08:41:29,962 - INFO - Deleting the FIP pool __fip_pool_public__
2025-11-12 08:41:30,116 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-11-12 08:41:30,122 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-11-12 08:41:30,122 - INFO - Deleting VN __public__
2025-11-12 08:41:30,260 - DEBUG - Requesting: http://10.0.0.22:8082/routing-instance/498ba4e6-4f18-40bb-a7f7-b001eede089f
2025-11-12 08:41:30,268 - DEBUG - Response Code: 404
2025-11-12 08:41:30,268 - DEBUG - Requesting: http://10.0.0.22:8082/domains
2025-11-12 08:41:30,273 - DEBUG - Requesting: http://10.0.0.22:8082/domain/6c3209c3-6307-4767-8fd9-1e60a49bc523
2025-11-12 08:41:30,284 - DEBUG - Requesting: http://10.0.0.22:8082/project/a044a930-430b-4898-8313-96cb56adc7b9
2025-11-12 08:41:30,338 - INFO - Validated that VN __public__ is not found in API Server
2025-11-12 08:41:30,338 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-11-12 08:41:30,344 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:41:30,355 - DEBUG - VN __public__ is not present in Agent 10.0.0.65
2025-11-12 08:41:30,355 - INFO - Validated that VN __public__ is not in any agent
2025-11-12 08:41:30,356 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:41:30,364 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-11-12 08:41:30,373 - DEBUG - Requesting: http://10.0.0.22:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-11-12 08:41:30,382 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-11-12 08:41:30,383 - INFO - Deleting pod default:ctest-nginx-pod-15544842
2025-11-12 08:41:30,394 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=c3b19062-5179-4729-99d6-75c23251d602
2025-11-12 08:41:30,415 - WARNING - Pod uuid c3b19062-5179-4729-99d6-75c23251d602 is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:35,415 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=c3b19062-5179-4729-99d6-75c23251d602
2025-11-12 08:41:35,423 - DEBUG - Pod c3b19062-5179-4729-99d6-75c23251d602 is not in agent 10.0.0.65 VM list
2025-11-12 08:41:35,424 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:412f37ee-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:41:35,429 - DEBUG - VMI 412f37ee-bfa3-11f0-bc61-fa163ea22c9e is removed from agent 10.0.0.65
2025-11-12 08:41:35,429 - INFO - Verified that pod ctest-nginx-pod-15544842 is removed in agent
2025-11-12 08:41:35,430 - INFO - Deleting pod default:ctest-nginx-pod-35328108
2025-11-12 08:41:35,455 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=6478989f-40c3-4b98-9d92-b9a9d292ace4
2025-11-12 08:41:35,467 - WARNING - Pod uuid 6478989f-40c3-4b98-9d92-b9a9d292ace4 is still seen in agent 10.0.0.65 VM list
2025-11-12 08:41:40,468 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=6478989f-40c3-4b98-9d92-b9a9d292ace4
2025-11-12 08:41:40,477 - DEBUG - Pod 6478989f-40c3-4b98-9d92-b9a9d292ace4 is not in agent 10.0.0.65 VM list
2025-11-12 08:41:40,477 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:402e9114-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:41:40,483 - DEBUG - VMI 402e9114-bfa3-11f0-bc61-fa163ea22c9e is removed from agent 10.0.0.65
2025-11-12 08:41:40,483 - INFO - Verified that pod ctest-nginx-pod-35328108 is removed in agent
2025-11-12 08:41:40,483 - INFO - Deleting service : ctest-nginx-svc-03825539
2025-11-12 08:41:40,723 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq?
2025-11-12 08:41:40,730 - DEBUG - No XMPP flaps were noticed during the test
2025-11-12 08:41:40,730 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:10]
2025-11-12 08:41:40,730 - INFO - --------------------------------------------------------------------------------
2025-11-12 08:41:47,163 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-11-12 08:41:47,291 - DEBUG - Output : cluster_project={}
2025-11-12 08:41:47,436 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-11-12 08:41:47,585 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-11-12 08:41:47,592 - DEBUG - [10.0.0.22]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-11-12 08:41:47,742 - DEBUG - Output : cluster_project={}
2025-11-12 08:41:57,833 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:41:57,912 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:41:57,950 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:42:03,052 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:42:03,100 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:42:08,188 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:42:08,220 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:42:08,296 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-11-12 08:42:08,546 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq?
2025-11-12 08:42:11,230 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_ApplicationPolicySetReq?
2025-11-12 08:42:11,235 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AclReq?
2025-11-12 08:42:11,258 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AclReq?
2025-11-12 08:42:13,282 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AclReq?
2025-11-12 08:45:22,330 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_AgentXmppConnectionStatusReq?
2025-11-12 08:45:22,394 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=8f6110d3-cf49-4533-af42-cc9b4a9f36cc
2025-11-12 08:45:22,414 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a28dc1e-8c82-4690-b277-38bd063029e4
2025-11-12 08:45:22,430 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:22,446 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=c255a8ad-b79f-4158-9dc1-ea9e05d150d8
2025-11-12 08:45:22,454 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=27fc7bbe-edc7-4914-9f13-218719ac0f68
2025-11-12 08:45:22,467 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:22,481 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:22,486 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:27,497 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=8f6110d3-cf49-4533-af42-cc9b4a9f36cc
2025-11-12 08:45:27,502 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6d83541a-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:27,516 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=c255a8ad-b79f-4158-9dc1-ea9e05d150d8
2025-11-12 08:45:27,518 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:27,520 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:27,522 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6e0672c8-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:27,523 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:27,525 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=27fc7bbe-edc7-4914-9f13-218719ac0f68
2025-11-12 08:45:27,525 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a28dc1e-8c82-4690-b277-38bd063029e4
2025-11-12 08:45:27,526 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:27,538 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6f3d139a-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:27,540 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6eb8afb0-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:32,530 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:32,532 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:32,537 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:32,540 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:37,541 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:37,543 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:37,544 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:37,546 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:42,555 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:42,557 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:42,558 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:42,559 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:47,570 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:47,572 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:47,573 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:47,574 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:52,590 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:52,591 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:52,593 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:52,594 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:57,605 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=5710d4a8-7525-4ae9-b0f5-bbc2f5dba0d8
2025-11-12 08:45:57,607 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=3a29cc30-71e7-434d-80d8-ce16f6aae76e
2025-11-12 08:45:57,608 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=05c1cf7b-ff07-408d-b29c-bca9541baa03
2025-11-12 08:45:57,610 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_VmListReq?uuid=e4819dae-edd9-4d36-8c07-ffb0573dd116
2025-11-12 08:45:57,620 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:70358a8e-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:57,623 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:6fc645fc-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:57,624 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:70a1e210-bfa3-11f0-bc61-fa163ea22c9e
2025-11-12 08:45:57,625 - DEBUG - Requesting: http://10.0.0.65:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:711bc6b6-bfa3-11f0-bc61-fa163ea22c9e