2025-05-09 22:06:04,504 - DEBUG - [10.0.0.48]: Running cmd : hostname
2025-05-09 22:06:04,750 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1221-1
2025-05-09 22:06:04,751 - DEBUG - [10.0.0.48]: Running cmd : hostname -f
2025-05-09 22:06:04,785 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1221-1
2025-05-09 22:06:04,785 - DEBUG - [10.0.0.48]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-05-09 22:06:04,848 - DEBUG - Output : NAMES
contrail_test_GfHn6jJvg
contrail-kubernetes-master-kubemanager-1
analytics_database-cassandra-1
analytics_database-nodemgr-1
analytics_database-query-engine-1
analytics_snmp-nodemgr-1
analytics_snmp-snmp-collector-1
analytics_snmp-topology-1
analytics_alarm-nodemgr-1
analytics_alarm-alarm-gen-1
analytics_alarm-kafka-1
analytics-collector-1
analytics-nodemgr-1
analytics-api-1
redis-redis-1
webui-job-1
webui-web-1
control-named-1
control-nodemgr-1
control-control-1
control-dns-1
config_database-rabbitmq-1
config_database-cassandra-1
config_database-zookeeper-1
config_database-nodemgr-1
config_api-devicemgr-1
config_api-svcmonitor-1
config_api-api-1
config_api-nodemgr-1
config_api-schema-1
config_api-dnsmasq-1
2025-05-09 22:06:04,849 - DEBUG - [10.0.0.48]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-05-09 22:06:04,882 - DEBUG - Output : 10.0.0.48/24
172.17.0.1/16
2025-05-09 22:06:04,882 - DEBUG - [10.0.0.48]: Running cmd : getent hosts 10.0.0.48 | head -n 1 | awk '{print $2}'
2025-05-09 22:06:04,954 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1221-1
2025-05-09 22:06:05,000 - DEBUG - [10.0.0.49]: Running cmd : hostname
2025-05-09 22:06:05,204 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1221-1
2025-05-09 22:06:05,204 - DEBUG - [10.0.0.49]: Running cmd : hostname -f
2025-05-09 22:06:05,228 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1221-1
2025-05-09 22:06:05,228 - DEBUG - [10.0.0.49]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-05-09 22:06:05,314 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-05-09 22:06:05,315 - DEBUG - [10.0.0.49]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2025-05-09 22:06:05,338 - DEBUG - Output : 10.0.0.49/24
2025-05-09 22:06:05,338 - DEBUG - [10.0.0.49]: Running cmd : getent hosts 10.0.0.49 | head -n 1 | awk '{print $2}'
2025-05-09 22:06:05,361 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1221-1
2025-05-09 22:06:05,442 - INFO - Using existing project ['default-domain', 'k8s-default'](73bb602f-388b-4439-97da-7dc807825ce0)
2025-05-09 22:06:05,528 - INFO - Using existing project ['default-domain', 'k8s-default'](73bb602f-388b-4439-97da-7dc807825ce0)
2025-05-09 22:06:05,708 - INFO - ================================================================================
2025-05-09 22:06:05,708 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-05-09 22:06:05,709 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-05-09 22:06:05,969 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-09 22:06:05,977 - DEBUG - Nothing to compare xmpp stats {'10.0.0.49': {'10.0.0.48': '0'}} with
2025-05-09 22:06:05,977 - INFO - Initial checks done. Running the testcase now
2025-05-09 22:06:05,977 - INFO -
2025-05-09 22:06:05,984 - INFO - Namespace default already exists
2025-05-09 22:06:05,984 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-05-09 22:06:06,155 - DEBUG - Output : cluster_project={}
2025-05-09 22:06:06,277 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-05-09 22:06:06,277 - DEBUG - Project uuid in Contrail is 73bb602f-388b-4439-97da-7dc807825ce0
2025-05-09 22:06:06,277 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-05-09 22:06:06,495 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-05-09 22:06:06,495 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_NamespaceDatabaseList?namespace_uuid=4513a350-1d43-4d5e-bd2c-60f8aa7a2a4c
2025-05-09 22:06:06,502 - INFO - Namespace default with uuid 4513a350-1d43-4d5e-bd2c-60f8aa7a2a4c found in kube manager
2025-05-09 22:06:06,502 - INFO - Namespace default verification passed
2025-05-09 22:06:06,505 - DEBUG - Service ctest-nginx-svc-36726259 not present
2025-05-09 22:06:06,506 - INFO - Creating service ctest-nginx-svc-36726259
2025-05-09 22:06:06,524 - INFO - Created Service ctest-nginx-svc-36726259
2025-05-09 22:06:06,527 - DEBUG - Pod ctest-nginx-pod-93895719 not present
2025-05-09 22:06:06,527 - INFO - Creating Pod ctest-nginx-pod-93895719
2025-05-09 22:06:06,571 - DEBUG - Node for Pod ctest-nginx-pod-93895719 not yet populated
2025-05-09 22:06:09,578 - DEBUG - Pod : ctest-nginx-pod-93895719 UUID is 795afa76-5ba5-4f23-a24f-b8769081fcf4
2025-05-09 22:06:09,580 - DEBUG - Pod ctest-nginx-pod-61933898 not present
2025-05-09 22:06:09,580 - INFO - Creating Pod ctest-nginx-pod-61933898
2025-05-09 22:06:09,605 - DEBUG - Pod : ctest-nginx-pod-61933898 UUID is ac1e73a2-886d-4b58-bcdd-776b42d0d774
2025-05-09 22:06:09,817 - INFO - Created VN __public__, UUID :ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,030 - DEBUG - Requesting: http://10.0.0.48:8082/domains
2025-05-09 22:06:10,035 - DEBUG - Requesting: http://10.0.0.48:8082/domain/9b656c15-bf45-43fd-8b9a-2c7e020332e9
2025-05-09 22:06:10,054 - DEBUG - Requesting: http://10.0.0.48:8082/project/73bb602f-388b-4439-97da-7dc807825ce0
2025-05-09 22:06:10,120 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,163 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,178 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/9576c640-6efd-4469-9858-95c2e2b43a40
2025-05-09 22:06:10,208 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/9576c640-6efd-4469-9858-95c2e2b43a40
2025-05-09 22:06:10,214 - DEBUG - Requesting: http://10.0.0.48:8082/route-target/3af82980-efa9-47c7-8c93-1e26c259a80e
2025-05-09 22:06:10,220 - DEBUG - Route Targets: ['target:64512:8000007']
2025-05-09 22:06:10,220 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,230 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/9576c640-6efd-4469-9858-95c2e2b43a40
2025-05-09 22:06:10,244 - INFO - Verified VN network id 13 for VN ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,244 - INFO - Verifications in API Server for VN __public__ passed
2025-05-09 22:06:10,244 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,253 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/9576c640-6efd-4469-9858-95c2e2b43a40
2025-05-09 22:06:10,259 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/9576c640-6efd-4469-9858-95c2e2b43a40
2025-05-09 22:06:10,268 - DEBUG - Requesting: http://10.0.0.48:8082/route-target/3af82980-efa9-47c7-8c93-1e26c259a80e
2025-05-09 22:06:10,274 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-05-09 22:06:10,306 - DEBUG - Control-node 10.0.0.48 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '17044616482447838174', 'uuid-lslong': '10505691877186122835'}, 'enable': 'true', 'created': '2025-05-09T22:06:09', 'last-modified': '2025-05-09T22:06:09', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.377032'}
2025-05-09 22:06:10,306 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:06:10,316 - DEBUG - Route Targets: ['target:64512:8000007']
2025-05-09 22:06:10,316 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-05-09 22:06:10,325 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-05-09 22:06:10,325 - DEBUG - Requesting: http://10.0.0.48:8082/domains
2025-05-09 22:06:10,337 - DEBUG - Requesting: http://10.0.0.48:8082/domain/9b656c15-bf45-43fd-8b9a-2c7e020332e9
2025-05-09 22:06:10,346 - DEBUG - Requesting: http://10.0.0.48:8082/project/73bb602f-388b-4439-97da-7dc807825ce0
2025-05-09 22:06:10,395 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,412 - DEBUG - =>VN __public__ has no policy to be verified
2025-05-09 22:06:10,412 - DEBUG - Verifying the vn in opserver
2025-05-09 22:06:10,412 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.48
2025-05-09 22:06:10,412 - DEBUG - Requesting: http://10.0.0.48:8081/analytics/uves/virtual-networks
2025-05-09 22:06:10,419 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.48:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-05-09 22:06:10,420 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-05-09 22:06:10,420 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:06:10,429 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:06:10,437 - DEBUG - Do not have enough data to verify VN in agent
2025-05-09 22:06:10,437 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:06:10,443 - DEBUG - VRF ids for VN __public__: {}
2025-05-09 22:06:10,545 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-05-09 22:06:10,648 - DEBUG - Requesting: http://10.0.0.48:8082/domains
2025-05-09 22:06:10,655 - DEBUG - Requesting: http://10.0.0.48:8082/domain/9b656c15-bf45-43fd-8b9a-2c7e020332e9
2025-05-09 22:06:10,662 - DEBUG - Requesting: http://10.0.0.48:8082/project/73bb602f-388b-4439-97da-7dc807825ce0
2025-05-09 22:06:10,704 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,714 - DEBUG - Requesting: http://10.0.0.48:8082/floating-ip-pool/18e66de9-f63c-4d8f-a44b-d2631ae5407f
2025-05-09 22:06:10,721 - DEBUG - Requesting: http://10.0.0.48:8082/domains
2025-05-09 22:06:10,725 - DEBUG - Requesting: http://10.0.0.48:8082/domain/9b656c15-bf45-43fd-8b9a-2c7e020332e9
2025-05-09 22:06:10,732 - DEBUG - Requesting: http://10.0.0.48:8082/project/73bb602f-388b-4439-97da-7dc807825ce0
2025-05-09 22:06:10,782 - DEBUG - Requesting: http://10.0.0.48:8082/virtual-network/ec8aa462-7112-4bde-91cb-b719cb62b453
2025-05-09 22:06:10,791 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-05-09 22:06:10,791 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-05-09 22:06:10,799 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-05-09 22:06:10,805 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-05-09 22:06:10,805 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-05-09 22:06:10,808 - DEBUG - Ingress ctest-nginx-ingress-12121932 not present
2025-05-09 22:06:10,809 - INFO - Creating Ingress ctest-nginx-ingress-12121932
2025-05-09 22:06:10,818 - INFO - Created Ingress ctest-nginx-ingress-12121932
2025-05-09 22:06:10,821 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-12121932
2025-05-09 22:06:13,826 - DEBUG - For Ingress ctest-nginx-ingress-12121932, Cluster IP: 10.47.255.249, External IPs ['74.209.124.252']
2025-05-09 22:06:13,826 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-12121932
2025-05-09 22:06:13,827 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=cead24b3-09a5-4dfb-85d1-cf878623e6bd
2025-05-09 22:06:13,832 - INFO - Ingress ctest-nginx-ingress-12121932 with uuid cead24b3-09a5-4dfb-85d1-cf878623e6bd found in kube manager
2025-05-09 22:06:13,833 - INFO - Ingress ctest-nginx-ingress-12121932 verification passed
2025-05-09 22:06:13,837 - DEBUG - Pod ctest-busybox-pod-43223230 not present
2025-05-09 22:06:13,837 - INFO - Creating Pod ctest-busybox-pod-43223230
2025-05-09 22:06:13,872 - DEBUG - Pod : ctest-busybox-pod-43223230 UUID is 278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:13,877 - DEBUG - Pod ctest-nginx-pod-93895719 not in running state.Currently in Pending
2025-05-09 22:06:18,891 - DEBUG - Pod ctest-nginx-pod-93895719 not in running state.Currently in Pending
2025-05-09 22:06:23,896 - INFO - Pod ctest-nginx-pod-93895719 is in running state.Got IP 10.47.255.251
2025-05-09 22:06:23,953 - DEBUG - Pod ctest-nginx-pod-93895719 has vmi cf1c9f16-2d21-11f0-9355-fa163e882dee
2025-05-09 22:06:23,953 - INFO - Verified pod ctest-nginx-pod-93895719 in contrail-api
2025-05-09 22:06:23,953 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:06:23,965 - DEBUG - VMI cf1c9f16-2d21-11f0-9355-fa163e882dee is active in agent 10.0.0.49
2025-05-09 22:06:23,965 - INFO - Verified Pod ctest-nginx-pod-93895719 in agent 10.0.0.49
2025-05-09 22:06:23,965 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_VirtualMachineDatabaseList?x=795afa76-5ba5-4f23-a24f-b8769081fcf4
2025-05-09 22:06:23,969 - INFO - Pod ctest-nginx-pod-93895719 with uuid 795afa76-5ba5-4f23-a24f-b8769081fcf4 found in kube manager
2025-05-09 22:06:23,969 - INFO - Pod ctest-nginx-pod-93895719 verification passed
2025-05-09 22:06:24,028 - DEBUG - [Pod ctest-nginx-pod-93895719] Cmd: echo ctest-nginx-pod-93895719 > /usr/share/nginx/html/index.html, Output:
2025-05-09 22:06:24,040 - DEBUG - Pod ctest-nginx-pod-61933898 not in running state.Currently in Pending
2025-05-09 22:06:29,046 - INFO - Pod ctest-nginx-pod-61933898 is in running state.Got IP 10.47.255.250
2025-05-09 22:06:29,059 - DEBUG - Pod ctest-nginx-pod-61933898 has vmi d02a15b4-2d21-11f0-9355-fa163e882dee
2025-05-09 22:06:29,059 - INFO - Verified pod ctest-nginx-pod-61933898 in contrail-api
2025-05-09 22:06:29,059 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:06:29,070 - DEBUG - VMI d02a15b4-2d21-11f0-9355-fa163e882dee is active in agent 10.0.0.49
2025-05-09 22:06:29,070 - INFO - Verified Pod ctest-nginx-pod-61933898 in agent 10.0.0.49
2025-05-09 22:06:29,070 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_VirtualMachineDatabaseList?x=ac1e73a2-886d-4b58-bcdd-776b42d0d774
2025-05-09 22:06:29,074 - INFO - Pod ctest-nginx-pod-61933898 with uuid ac1e73a2-886d-4b58-bcdd-776b42d0d774 found in kube manager
2025-05-09 22:06:29,074 - INFO - Pod ctest-nginx-pod-61933898 verification passed
2025-05-09 22:06:29,127 - DEBUG - [Pod ctest-nginx-pod-61933898] Cmd: echo ctest-nginx-pod-61933898 > /usr/share/nginx/html/index.html, Output:
2025-05-09 22:06:29,132 - INFO - Pod ctest-busybox-pod-43223230 is in running state.Got IP 10.47.255.248
2025-05-09 22:06:29,147 - DEBUG - Pod ctest-busybox-pod-43223230 has vmi d2b668fa-2d21-11f0-9355-fa163e882dee
2025-05-09 22:06:29,147 - INFO - Verified pod ctest-busybox-pod-43223230 in contrail-api
2025-05-09 22:06:29,147 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:06:29,161 - DEBUG - VMI d2b668fa-2d21-11f0-9355-fa163e882dee is active in agent 10.0.0.49
2025-05-09 22:06:29,161 - INFO - Verified Pod ctest-busybox-pod-43223230 in agent 10.0.0.49
2025-05-09 22:06:29,161 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_VirtualMachineDatabaseList?x=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:29,166 - INFO - Pod ctest-busybox-pod-43223230 with uuid 278a429d-1891-4981-bb14-ed207347649d found in kube manager
2025-05-09 22:06:29,166 - INFO - Pod ctest-busybox-pod-43223230 verification passed
2025-05-09 22:06:29,221 - DEBUG - [Pod ctest-busybox-pod-43223230] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-61933898
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-09 22:06:29,221 - DEBUG - [Pod Pod ctest-busybox-pod-43223230] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-05-09 22:06:29,221 - DEBUG - [Pod Pod ctest-busybox-pod-43223230] Cmd output: ctest-nginx-pod-61933898
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-09 22:06:29,270 - DEBUG - [Pod ctest-busybox-pod-43223230] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-61933898
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-09 22:06:29,271 - DEBUG - [Pod Pod ctest-busybox-pod-43223230] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-05-09 22:06:29,271 - DEBUG - [Pod Pod ctest-busybox-pod-43223230] Cmd output: ctest-nginx-pod-61933898
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-09 22:06:29,333 - DEBUG - [Pod ctest-busybox-pod-43223230] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-93895719
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-09 22:06:29,333 - DEBUG - [Pod Pod ctest-busybox-pod-43223230] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-05-09 22:06:29,333 - DEBUG - [Pod Pod ctest-busybox-pod-43223230] Cmd output: ctest-nginx-pod-93895719
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-05-09 22:06:29,333 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-93895719': 1, 'ctest-nginx-pod-61933898': 2}
2025-05-09 22:06:29,333 - WARNING - Nothing to delete parallely
2025-05-09 22:06:29,334 - INFO - Deleting pod default:ctest-busybox-pod-43223230
2025-05-09 22:06:29,344 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:29,350 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:06:34,351 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:34,358 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:06:39,359 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:39,370 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:06:44,371 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:44,379 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:06:49,379 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:49,389 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:06:54,390 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:54,397 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:06:59,398 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:06:59,406 - WARNING - Pod uuid 278a429d-1891-4981-bb14-ed207347649d is still seen in agent 10.0.0.49 VM list
2025-05-09 22:07:04,407 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=278a429d-1891-4981-bb14-ed207347649d
2025-05-09 22:07:04,413 - DEBUG - Pod 278a429d-1891-4981-bb14-ed207347649d is not in agent 10.0.0.49 VM list
2025-05-09 22:07:04,413 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d2b668fa-2d21-11f0-9355-fa163e882dee
2025-05-09 22:07:04,420 - DEBUG - VMI d2b668fa-2d21-11f0-9355-fa163e882dee is removed from agent 10.0.0.49
2025-05-09 22:07:04,420 - INFO - Verified that pod ctest-busybox-pod-43223230 is removed in agent
2025-05-09 22:07:04,420 - INFO - Deleting Ingress : ctest-nginx-ingress-12121932
2025-05-09 22:07:04,431 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=cead24b3-09a5-4dfb-85d1-cf878623e6bd
2025-05-09 22:07:04,437 - ERROR - Ingress ctest-nginx-ingress-12121932 with uuid cead24b3-09a5-4dfb-85d1-cf878623e6bd still found in kube manager
2025-05-09 22:07:05,438 - DEBUG - Requesting: http://10.0.0.48:8108/Snh_LoadbalancerDatabaseList?x=cead24b3-09a5-4dfb-85d1-cf878623e6bd
2025-05-09 22:07:05,443 - INFO - Ingress ctest-nginx-ingress-12121932 with uuid cead24b3-09a5-4dfb-85d1-cf878623e6bd deleted successfully from kube manager
2025-05-09 22:07:05,443 - INFO - Verifications on Ingress %s cleanup passed
2025-05-09 22:07:05,443 - INFO - Deleting the FIP pool __fip_pool_public__
2025-05-09 22:07:05,600 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-05-09 22:07:05,608 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-05-09 22:07:05,608 - INFO - Deleting VN __public__
2025-05-09 22:07:05,776 - DEBUG - Requesting: http://10.0.0.48:8082/routing-instance/9576c640-6efd-4469-9858-95c2e2b43a40
2025-05-09 22:07:05,795 - DEBUG - Response Code: 404
2025-05-09 22:07:05,795 - DEBUG - Requesting: http://10.0.0.48:8082/domains
2025-05-09 22:07:05,832 - DEBUG - Requesting: http://10.0.0.48:8082/domain/9b656c15-bf45-43fd-8b9a-2c7e020332e9
2025-05-09 22:07:05,850 - DEBUG - Requesting: http://10.0.0.48:8082/project/73bb602f-388b-4439-97da-7dc807825ce0
2025-05-09 22:07:05,910 - INFO - Validated that VN __public__ is not found in API Server
2025-05-09 22:07:05,910 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-05-09 22:07:05,919 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:07:05,934 - DEBUG - VN __public__ is not present in Agent 10.0.0.49
2025-05-09 22:07:05,934 - INFO - Validated that VN __public__ is not in any agent
2025-05-09 22:07:05,934 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:07:05,940 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-05-09 22:07:05,946 - DEBUG - Requesting: http://10.0.0.48:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-05-09 22:07:05,951 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-05-09 22:07:05,952 - INFO - Deleting pod default:ctest-nginx-pod-61933898
2025-05-09 22:07:05,968 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=ac1e73a2-886d-4b58-bcdd-776b42d0d774
2025-05-09 22:07:05,981 - WARNING - Pod uuid ac1e73a2-886d-4b58-bcdd-776b42d0d774 is still seen in agent 10.0.0.49 VM list
2025-05-09 22:07:10,981 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=ac1e73a2-886d-4b58-bcdd-776b42d0d774
2025-05-09 22:07:10,989 - DEBUG - Pod ac1e73a2-886d-4b58-bcdd-776b42d0d774 is not in agent 10.0.0.49 VM list
2025-05-09 22:07:10,989 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d02a15b4-2d21-11f0-9355-fa163e882dee
2025-05-09 22:07:10,995 - DEBUG - VMI d02a15b4-2d21-11f0-9355-fa163e882dee is removed from agent 10.0.0.49
2025-05-09 22:07:10,995 - INFO - Verified that pod ctest-nginx-pod-61933898 is removed in agent
2025-05-09 22:07:10,995 - INFO - Deleting pod default:ctest-nginx-pod-93895719
2025-05-09 22:07:11,008 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=795afa76-5ba5-4f23-a24f-b8769081fcf4
2025-05-09 22:07:11,025 - WARNING - Pod uuid 795afa76-5ba5-4f23-a24f-b8769081fcf4 is still seen in agent 10.0.0.49 VM list
2025-05-09 22:07:16,027 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=795afa76-5ba5-4f23-a24f-b8769081fcf4
2025-05-09 22:07:16,033 - DEBUG - Pod 795afa76-5ba5-4f23-a24f-b8769081fcf4 is not in agent 10.0.0.49 VM list
2025-05-09 22:07:16,033 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cf1c9f16-2d21-11f0-9355-fa163e882dee
2025-05-09 22:07:16,039 - DEBUG - VMI cf1c9f16-2d21-11f0-9355-fa163e882dee is removed from agent 10.0.0.49
2025-05-09 22:07:16,039 - INFO - Verified that pod ctest-nginx-pod-93895719 is removed in agent
2025-05-09 22:07:16,039 - INFO - Deleting service : ctest-nginx-svc-36726259
2025-05-09 22:07:16,302 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-09 22:07:16,309 - DEBUG - No XMPP flaps were noticed during the test
2025-05-09 22:07:16,309 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:11]
2025-05-09 22:07:16,309 - INFO - --------------------------------------------------------------------------------
2025-05-09 22:07:22,725 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-05-09 22:07:22,887 - DEBUG - Output : cluster_project={}
2025-05-09 22:07:22,946 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-05-09 22:07:23,134 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-05-09 22:07:23,143 - DEBUG - [10.0.0.48]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-05-09 22:07:23,343 - DEBUG - Output : cluster_project={}
2025-05-09 22:07:38,500 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:38,536 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:38,618 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:38,697 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:43,740 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:43,779 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:44,014 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:44,051 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-05-09 22:07:44,292 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-09 22:07:46,824 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_ApplicationPolicySetReq?
2025-05-09 22:07:46,831 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AclReq?
2025-05-09 22:07:46,875 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AclReq?
2025-05-09 22:07:48,906 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AclReq?
2025-05-09 22:10:48,963 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_AgentXmppConnectionStatusReq?
2025-05-09 22:10:49,007 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=29074487-6869-4496-bdad-c225e7311084
2025-05-09 22:10:49,018 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=970b5288-2c8b-4902-aba3-76cae081ae4d
2025-05-09 22:10:49,033 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:10:49,041 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=02cb5bb2-926a-4eb7-9219-51834511bb3e
2025-05-09 22:10:49,045 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:10:49,050 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:10:49,054 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:10:49,062 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6c41e49e-aad4-4bab-9766-6d290a5326af
2025-05-09 22:10:54,070 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=29074487-6869-4496-bdad-c225e7311084
2025-05-09 22:10:54,072 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=970b5288-2c8b-4902-aba3-76cae081ae4d
2025-05-09 22:10:54,081 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:fc89229e-2d21-11f0-9355-fa163e882dee
2025-05-09 22:10:54,084 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:fd0410ee-2d21-11f0-9355-fa163e882dee
2025-05-09 22:10:54,101 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=6c41e49e-aad4-4bab-9766-6d290a5326af
2025-05-09 22:10:54,104 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:10:54,105 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:10:54,106 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:10:54,107 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:10:54,109 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=02cb5bb2-926a-4eb7-9219-51834511bb3e
2025-05-09 22:10:54,112 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:fe17f9c8-2d21-11f0-9355-fa163e882dee
2025-05-09 22:10:54,130 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:fd9e729c-2d21-11f0-9355-fa163e882dee
2025-05-09 22:10:59,128 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:10:59,133 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:10:59,135 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:10:59,137 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:11:04,142 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:11:04,150 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:11:04,151 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:11:04,153 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:11:09,149 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:11:09,163 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:11:09,165 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:11:09,166 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:11:14,158 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:11:14,177 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:11:14,179 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:11:14,180 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:11:19,166 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:11:19,191 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:11:19,193 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:11:19,194 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:11:24,185 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=a08ed947-8305-4fe8-a842-0c4377e64425
2025-05-09 22:11:24,192 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ff88b676-2d21-11f0-9355-fa163e882dee
2025-05-09 22:11:24,205 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=477b5470-023f-466c-99a4-4eb90e6794d0
2025-05-09 22:11:24,207 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=beedcf6e-7a81-4ace-b6b9-ccffd8a1fcd8
2025-05-09 22:11:24,208 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_VmListReq?uuid=e40317a1-ee6f-4cce-bf92-4fe7a347dd42
2025-05-09 22:11:24,218 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:fff86ed0-2d21-11f0-9355-fa163e882dee
2025-05-09 22:11:24,221 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ff186cfe-2d21-11f0-9355-fa163e882dee
2025-05-09 22:11:24,222 - DEBUG - Requesting: http://10.0.0.49:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:fea48b40-2d21-11f0-9355-fa163e882dee