2025-08-12 01:35:19,111 - DEBUG - [10.0.0.38]: Running cmd : hostname
2025-08-12 01:35:19,355 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1375-1
2025-08-12 01:35:19,356 - DEBUG - [10.0.0.38]: Running cmd : hostname -f
2025-08-12 01:35:19,389 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1375-1
2025-08-12 01:35:19,389 - DEBUG - [10.0.0.38]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-12 01:35:19,442 - DEBUG - Output : NAMES
contrail_test_gfGFOOg0P
contrail-kubernetes-master-kubemanager-1
webui-job-1
webui-web-1
analytics_database-query-engine-1
analytics_database-cassandra-1
analytics_database-nodemgr-1
analytics_snmp-nodemgr-1
analytics_snmp-topology-1
analytics_snmp-snmp-collector-1
analytics_alarm-alarm-gen-1
analytics_alarm-kafka-1
analytics_alarm-nodemgr-1
analytics-collector-1
analytics-nodemgr-1
analytics-api-1
redis-redis-1
control-named-1
control-control-1
control-nodemgr-1
control-dns-1
config_database-zookeeper-1
config_database-nodemgr-1
config_database-rabbitmq-1
config_database-cassandra-1
config_api-devicemgr-1
config_api-svcmonitor-1
config_api-dnsmasq-1
config_api-api-1
config_api-schema-1
config_api-nodemgr-1
2025-08-12 01:35:19,443 - DEBUG - [10.0.0.38]: Running cmd : ip -4 -o addr show | awk '{print $4}'
2025-08-12 01:35:19,518 - DEBUG - Output : 127.0.0.1/8
10.0.0.38/24
252.38.0.1/8
10.204.175.1/24
172.17.0.1/16
2025-08-12 01:35:19,518 - DEBUG - [10.0.0.38]: Running cmd : getent hosts 10.0.0.38 | head -n 1 | awk '{print $2}'
2025-08-12 01:35:19,552 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-1375-1
2025-08-12 01:35:19,640 - DEBUG - [10.0.0.249]: Running cmd : hostname
2025-08-12 01:35:19,836 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1375-1
2025-08-12 01:35:19,836 - DEBUG - [10.0.0.249]: Running cmd : hostname -f
2025-08-12 01:35:19,859 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1375-1
2025-08-12 01:35:19,859 - DEBUG - [10.0.0.249]: Running cmd : docker ps 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2025-08-12 01:35:19,902 - DEBUG - Output : NAMES
vrouter-vrouter-agent-1
vrouter-nodemgr-1
2025-08-12 01:35:19,902 - DEBUG - [10.0.0.249]: Running cmd : ip -4 -o addr show dev vhost0 | awk '{print $4}'
2025-08-12 01:35:19,958 - DEBUG - Output : 10.0.0.249/24
2025-08-12 01:35:19,958 - DEBUG - [10.0.0.249]: Running cmd : getent hosts 10.0.0.249 | head -n 1 | awk '{print $2}'
2025-08-12 01:35:19,982 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-1375-1
2025-08-12 01:35:20,048 - INFO - Using existing project ['default-domain', 'k8s-default'](04f09fb4-3420-448a-b645-e1ec2b88c8ee)
2025-08-12 01:35:20,144 - INFO - Using existing project ['default-domain', 'k8s-default'](04f09fb4-3420-448a-b645-e1ec2b88c8ee)
2025-08-12 01:35:20,278 - INFO - ================================================================================
2025-08-12 01:35:20,278 - INFO - STARTING TEST : test_ingress_ip_assignment
2025-08-12 01:35:20,279 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2025-08-12 01:35:20,490 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-12 01:35:20,497 - DEBUG - Nothing to compare xmpp stats {'10.0.0.249': {'10.0.0.38': '0'}} with
2025-08-12 01:35:20,497 - INFO - Initial checks done. Running the testcase now
2025-08-12 01:35:20,497 - INFO -
2025-08-12 01:35:20,501 - INFO - Namespace default already exists
2025-08-12 01:35:20,501 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-08-12 01:35:20,616 - DEBUG - Output : cluster_project={}
2025-08-12 01:35:20,665 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2025-08-12 01:35:20,665 - DEBUG - Project uuid in Contrail is 04f09fb4-3420-448a-b645-e1ec2b88c8ee
2025-08-12 01:35:20,665 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-08-12 01:35:20,795 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-08-12 01:35:20,795 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_NamespaceDatabaseList?namespace_uuid=159e06e6-97e0-4504-863b-daf65ffdd695
2025-08-12 01:35:20,800 - INFO - Namespace default with uuid 159e06e6-97e0-4504-863b-daf65ffdd695 found in kube manager
2025-08-12 01:35:20,801 - INFO - Namespace default verification passed
2025-08-12 01:35:20,803 - DEBUG - Service ctest-nginx-svc-20323829 not present
2025-08-12 01:35:20,804 - INFO - Creating service ctest-nginx-svc-20323829
2025-08-12 01:35:20,828 - INFO - Created Service ctest-nginx-svc-20323829
2025-08-12 01:35:20,830 - DEBUG - Pod ctest-nginx-pod-29932836 not present
2025-08-12 01:35:20,831 - INFO - Creating Pod ctest-nginx-pod-29932836
2025-08-12 01:35:20,878 - DEBUG - Node for Pod ctest-nginx-pod-29932836 not yet populated
2025-08-12 01:35:23,885 - DEBUG - Pod : ctest-nginx-pod-29932836 UUID is 70d04a4d-deb0-4b6c-b518-abcffe127cc3
2025-08-12 01:35:23,888 - DEBUG - Pod ctest-nginx-pod-88258945 not present
2025-08-12 01:35:23,888 - INFO - Creating Pod ctest-nginx-pod-88258945
2025-08-12 01:35:23,908 - DEBUG - Pod : ctest-nginx-pod-88258945 UUID is c3a3e1c7-702d-4ba5-9318-48ff59ec8c05
2025-08-12 01:35:24,121 - INFO - Created VN __public__, UUID :9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,352 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-12 01:35:24,361 - DEBUG - Requesting: http://10.0.0.38:8082/domain/d289bb91-5680-40e3-bba1-181e9a2c7f96
2025-08-12 01:35:24,386 - DEBUG - Requesting: http://10.0.0.38:8082/project/04f09fb4-3420-448a-b645-e1ec2b88c8ee
2025-08-12 01:35:24,469 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,496 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,504 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/912c795c-1a7c-4024-8315-936419fae136
2025-08-12 01:35:24,511 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/912c795c-1a7c-4024-8315-936419fae136
2025-08-12 01:35:24,517 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/fd6c3c45-efab-4dd7-b825-cc6fd6fe0434
2025-08-12 01:35:24,522 - DEBUG - Route Targets: ['target:64512:8000007']
2025-08-12 01:35:24,522 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,530 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/912c795c-1a7c-4024-8315-936419fae136
2025-08-12 01:35:24,544 - INFO - Verified VN network id 13 for VN 9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,544 - INFO - Verifications in API Server for VN __public__ passed
2025-08-12 01:35:24,544 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,554 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/912c795c-1a7c-4024-8315-936419fae136
2025-08-12 01:35:24,559 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/912c795c-1a7c-4024-8315-936419fae136
2025-08-12 01:35:24,585 - DEBUG - Requesting: http://10.0.0.38:8082/route-target/fd6c3c45-efab-4dd7-b825-cc6fd6fe0434
2025-08-12 01:35:24,592 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-08-12 01:35:24,598 - DEBUG - Control-node 10.0.0.38 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '10895514081402047420', 'uuid-lslong': '12110073170717116725'}, 'enable': 'true', 'created': '2025-08-12T01:35:24', 'last-modified': '2025-08-12T01:35:24', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.330468'}
2025-08-12 01:35:24,598 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:35:24,606 - DEBUG - Route Targets: ['target:64512:8000007']
2025-08-12 01:35:24,606 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2025-08-12 01:35:24,607 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2025-08-12 01:35:24,607 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-12 01:35:24,612 - DEBUG - Requesting: http://10.0.0.38:8082/domain/d289bb91-5680-40e3-bba1-181e9a2c7f96
2025-08-12 01:35:24,619 - DEBUG - Requesting: http://10.0.0.38:8082/project/04f09fb4-3420-448a-b645-e1ec2b88c8ee
2025-08-12 01:35:24,667 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,676 - DEBUG - =>VN __public__ has no policy to be verified
2025-08-12 01:35:24,676 - DEBUG - Verifying the vn in opserver
2025-08-12 01:35:24,676 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.38
2025-08-12 01:35:24,676 - DEBUG - Requesting: http://10.0.0.38:8081/analytics/uves/virtual-networks
2025-08-12 01:35:24,681 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.38:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2025-08-12 01:35:24,681 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2025-08-12 01:35:24,682 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:35:24,691 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:35:24,696 - DEBUG - Do not have enough data to verify VN in agent
2025-08-12 01:35:24,696 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:35:24,702 - DEBUG - VRF ids for VN __public__: {}
2025-08-12 01:35:24,804 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2025-08-12 01:35:24,914 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-12 01:35:24,919 - DEBUG - Requesting: http://10.0.0.38:8082/domain/d289bb91-5680-40e3-bba1-181e9a2c7f96
2025-08-12 01:35:24,926 - DEBUG - Requesting: http://10.0.0.38:8082/project/04f09fb4-3420-448a-b645-e1ec2b88c8ee
2025-08-12 01:35:24,961 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:24,970 - DEBUG - Requesting: http://10.0.0.38:8082/floating-ip-pool/715ee776-547c-48d1-b9e2-cb1427e93f48
2025-08-12 01:35:24,976 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-12 01:35:24,980 - DEBUG - Requesting: http://10.0.0.38:8082/domain/d289bb91-5680-40e3-bba1-181e9a2c7f96
2025-08-12 01:35:24,987 - DEBUG - Requesting: http://10.0.0.38:8082/project/04f09fb4-3420-448a-b645-e1ec2b88c8ee
2025-08-12 01:35:25,024 - DEBUG - Requesting: http://10.0.0.38:8082/virtual-network/9734a454-833c-4bbc-a80f-9f6308950935
2025-08-12 01:35:25,033 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2025-08-12 01:35:25,033 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2025-08-12 01:35:25,039 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-08-12 01:35:25,045 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2025-08-12 01:35:25,045 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2025-08-12 01:35:25,048 - DEBUG - Ingress ctest-nginx-ingress-24302260 not present
2025-08-12 01:35:25,049 - INFO - Creating Ingress ctest-nginx-ingress-24302260
2025-08-12 01:35:25,054 - INFO - Created Ingress ctest-nginx-ingress-24302260
2025-08-12 01:35:25,057 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-24302260
2025-08-12 01:35:28,061 - DEBUG - For Ingress ctest-nginx-ingress-24302260, Cluster IP: 10.47.255.249, External IPs ['51.130.49.124']
2025-08-12 01:35:28,061 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-24302260
2025-08-12 01:35:28,061 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=0e760c60-5bb9-4496-8d9a-737a6c8c2fec
2025-08-12 01:35:28,066 - INFO - Ingress ctest-nginx-ingress-24302260 with uuid 0e760c60-5bb9-4496-8d9a-737a6c8c2fec found in kube manager
2025-08-12 01:35:28,066 - INFO - Ingress ctest-nginx-ingress-24302260 verification passed
2025-08-12 01:35:28,069 - DEBUG - Pod ctest-busybox-pod-26427225 not present
2025-08-12 01:35:28,070 - INFO - Creating Pod ctest-busybox-pod-26427225
2025-08-12 01:35:28,092 - DEBUG - Pod : ctest-busybox-pod-26427225 UUID is 8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:35:28,098 - DEBUG - Pod ctest-nginx-pod-29932836 not in running state.Currently in Pending
2025-08-12 01:35:33,108 - DEBUG - Pod ctest-nginx-pod-29932836 not in running state.Currently in Pending
2025-08-12 01:35:38,113 - DEBUG - Pod ctest-nginx-pod-29932836 not in running state.Currently in Pending
2025-08-12 01:35:43,125 - INFO - Pod ctest-nginx-pod-29932836 is in running state.Got IP 10.47.255.251
2025-08-12 01:35:43,184 - DEBUG - Pod ctest-nginx-pod-29932836 has vmi 9d0a20c6-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:35:43,185 - INFO - Verified pod ctest-nginx-pod-29932836 in contrail-api
2025-08-12 01:35:43,185 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:35:43,196 - DEBUG - VMI 9d0a20c6-771c-11f0-bea2-fa163e5be5c4 is active in agent 10.0.0.249
2025-08-12 01:35:43,196 - INFO - Verified Pod ctest-nginx-pod-29932836 in agent 10.0.0.249
2025-08-12 01:35:43,197 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=70d04a4d-deb0-4b6c-b518-abcffe127cc3
2025-08-12 01:35:43,201 - INFO - Pod ctest-nginx-pod-29932836 with uuid 70d04a4d-deb0-4b6c-b518-abcffe127cc3 found in kube manager
2025-08-12 01:35:43,201 - INFO - Pod ctest-nginx-pod-29932836 verification passed
2025-08-12 01:35:43,253 - DEBUG - [Pod ctest-nginx-pod-29932836] Cmd: echo ctest-nginx-pod-29932836 > /usr/share/nginx/html/index.html, Output:
2025-08-12 01:35:43,259 - DEBUG - Pod ctest-nginx-pod-88258945 not in running state.Currently in Pending
2025-08-12 01:35:48,265 - INFO - Pod ctest-nginx-pod-88258945 is in running state.Got IP 10.47.255.250
2025-08-12 01:35:48,277 - DEBUG - Pod ctest-nginx-pod-88258945 has vmi 9def9b38-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:35:48,277 - INFO - Verified pod ctest-nginx-pod-88258945 in contrail-api
2025-08-12 01:35:48,277 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:35:48,289 - DEBUG - VMI 9def9b38-771c-11f0-bea2-fa163e5be5c4 is active in agent 10.0.0.249
2025-08-12 01:35:48,289 - INFO - Verified Pod ctest-nginx-pod-88258945 in agent 10.0.0.249
2025-08-12 01:35:48,289 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=c3a3e1c7-702d-4ba5-9318-48ff59ec8c05
2025-08-12 01:35:48,293 - INFO - Pod ctest-nginx-pod-88258945 with uuid c3a3e1c7-702d-4ba5-9318-48ff59ec8c05 found in kube manager
2025-08-12 01:35:48,294 - INFO - Pod ctest-nginx-pod-88258945 verification passed
2025-08-12 01:35:48,344 - DEBUG - [Pod ctest-nginx-pod-88258945] Cmd: echo ctest-nginx-pod-88258945 > /usr/share/nginx/html/index.html, Output:
2025-08-12 01:35:48,349 - INFO - Pod ctest-busybox-pod-26427225 is in running state.Got IP 10.47.255.248
2025-08-12 01:35:48,361 - DEBUG - Pod ctest-busybox-pod-26427225 has vmi a06c8182-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:35:48,361 - INFO - Verified pod ctest-busybox-pod-26427225 in contrail-api
2025-08-12 01:35:48,361 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:35:48,370 - DEBUG - VMI a06c8182-771c-11f0-bea2-fa163e5be5c4 is active in agent 10.0.0.249
2025-08-12 01:35:48,371 - INFO - Verified Pod ctest-busybox-pod-26427225 in agent 10.0.0.249
2025-08-12 01:35:48,371 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_VirtualMachineDatabaseList?x=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:35:48,375 - INFO - Pod ctest-busybox-pod-26427225 with uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 found in kube manager
2025-08-12 01:35:48,375 - INFO - Pod ctest-busybox-pod-26427225 verification passed
2025-08-12 01:35:48,427 - DEBUG - [Pod ctest-busybox-pod-26427225] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-29932836
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-12 01:35:48,427 - DEBUG - [Pod Pod ctest-busybox-pod-26427225] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-08-12 01:35:48,427 - DEBUG - [Pod Pod ctest-busybox-pod-26427225] Cmd output: ctest-nginx-pod-29932836
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-12 01:35:48,479 - DEBUG - [Pod ctest-busybox-pod-26427225] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-88258945
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-12 01:35:48,479 - DEBUG - [Pod Pod ctest-busybox-pod-26427225] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2025-08-12 01:35:48,479 - DEBUG - [Pod Pod ctest-busybox-pod-26427225] Cmd output: ctest-nginx-pod-88258945
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2025-08-12 01:35:48,479 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-29932836': 1, 'ctest-nginx-pod-88258945': 1}
2025-08-12 01:35:48,480 - WARNING - Nothing to delete parallely
2025-08-12 01:35:48,480 - INFO - Deleting pod default:ctest-busybox-pod-26427225
2025-08-12 01:35:48,502 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:35:48,509 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:35:53,510 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:35:53,519 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:35:58,520 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:35:58,527 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:03,528 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:36:03,534 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:08,535 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:36:08,542 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:13,543 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:36:13,551 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:18,552 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:36:18,559 - WARNING - Pod uuid 8d3b9e66-c515-4616-8e8a-c3324899ba00 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:23,560 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=8d3b9e66-c515-4616-8e8a-c3324899ba00
2025-08-12 01:36:23,566 - DEBUG - Pod 8d3b9e66-c515-4616-8e8a-c3324899ba00 is not in agent 10.0.0.249 VM list
2025-08-12 01:36:23,566 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:a06c8182-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:36:23,571 - DEBUG - VMI a06c8182-771c-11f0-bea2-fa163e5be5c4 is removed from agent 10.0.0.249
2025-08-12 01:36:23,571 - INFO - Verified that pod ctest-busybox-pod-26427225 is removed in agent
2025-08-12 01:36:23,571 - INFO - Deleting Ingress : ctest-nginx-ingress-24302260
2025-08-12 01:36:23,576 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=0e760c60-5bb9-4496-8d9a-737a6c8c2fec
2025-08-12 01:36:23,581 - ERROR - Ingress ctest-nginx-ingress-24302260 with uuid 0e760c60-5bb9-4496-8d9a-737a6c8c2fec still found in kube manager
2025-08-12 01:36:24,581 - DEBUG - Requesting: http://10.0.0.38:8108/Snh_LoadbalancerDatabaseList?x=0e760c60-5bb9-4496-8d9a-737a6c8c2fec
2025-08-12 01:36:24,585 - INFO - Ingress ctest-nginx-ingress-24302260 with uuid 0e760c60-5bb9-4496-8d9a-737a6c8c2fec deleted successfully from kube manager
2025-08-12 01:36:24,586 - INFO - Verifications on Ingress %s cleanup passed
2025-08-12 01:36:24,586 - INFO - Deleting the FIP pool __fip_pool_public__
2025-08-12 01:36:24,718 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2025-08-12 01:36:24,729 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2025-08-12 01:36:24,729 - INFO - Deleting VN __public__
2025-08-12 01:36:24,880 - DEBUG - Requesting: http://10.0.0.38:8082/routing-instance/912c795c-1a7c-4024-8315-936419fae136
2025-08-12 01:36:24,885 - DEBUG - Response Code: 404
2025-08-12 01:36:24,885 - DEBUG - Requesting: http://10.0.0.38:8082/domains
2025-08-12 01:36:24,897 - DEBUG - Requesting: http://10.0.0.38:8082/domain/d289bb91-5680-40e3-bba1-181e9a2c7f96
2025-08-12 01:36:24,906 - DEBUG - Requesting: http://10.0.0.38:8082/project/04f09fb4-3420-448a-b645-e1ec2b88c8ee
2025-08-12 01:36:24,938 - INFO - Validated that VN __public__ is not found in API Server
2025-08-12 01:36:24,938 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2025-08-12 01:36:24,946 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:36:24,954 - DEBUG - VN __public__ is not present in Agent 10.0.0.249
2025-08-12 01:36:24,954 - INFO - Validated that VN __public__ is not in any agent
2025-08-12 01:36:24,954 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:36:24,963 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2025-08-12 01:36:24,967 - DEBUG - Requesting: http://10.0.0.38:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2025-08-12 01:36:24,972 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2025-08-12 01:36:24,972 - INFO - Deleting pod default:ctest-nginx-pod-88258945
2025-08-12 01:36:24,986 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c3a3e1c7-702d-4ba5-9318-48ff59ec8c05
2025-08-12 01:36:25,005 - WARNING - Pod uuid c3a3e1c7-702d-4ba5-9318-48ff59ec8c05 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:30,006 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=c3a3e1c7-702d-4ba5-9318-48ff59ec8c05
2025-08-12 01:36:30,012 - DEBUG - Pod c3a3e1c7-702d-4ba5-9318-48ff59ec8c05 is not in agent 10.0.0.249 VM list
2025-08-12 01:36:30,013 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9def9b38-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:36:30,018 - DEBUG - VMI 9def9b38-771c-11f0-bea2-fa163e5be5c4 is removed from agent 10.0.0.249
2025-08-12 01:36:30,018 - INFO - Verified that pod ctest-nginx-pod-88258945 is removed in agent
2025-08-12 01:36:30,018 - INFO - Deleting pod default:ctest-nginx-pod-29932836
2025-08-12 01:36:30,035 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=70d04a4d-deb0-4b6c-b518-abcffe127cc3
2025-08-12 01:36:30,049 - WARNING - Pod uuid 70d04a4d-deb0-4b6c-b518-abcffe127cc3 is still seen in agent 10.0.0.249 VM list
2025-08-12 01:36:35,050 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=70d04a4d-deb0-4b6c-b518-abcffe127cc3
2025-08-12 01:36:35,057 - DEBUG - Pod 70d04a4d-deb0-4b6c-b518-abcffe127cc3 is not in agent 10.0.0.249 VM list
2025-08-12 01:36:35,057 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:9d0a20c6-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:36:35,062 - DEBUG - VMI 9d0a20c6-771c-11f0-bea2-fa163e5be5c4 is removed from agent 10.0.0.249
2025-08-12 01:36:35,062 - INFO - Verified that pod ctest-nginx-pod-29932836 is removed in agent
2025-08-12 01:36:35,062 - INFO - Deleting service : ctest-nginx-svc-20323829
2025-08-12 01:36:35,239 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-12 01:36:35,246 - DEBUG - No XMPP flaps were noticed during the test
2025-08-12 01:36:35,247 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:15]
2025-08-12 01:36:35,247 - INFO - --------------------------------------------------------------------------------
2025-08-12 01:36:47,640 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-08-12 01:36:47,756 - DEBUG - Output : cluster_project={}
2025-08-12 01:36:47,800 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2025-08-12 01:36:47,992 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2025-08-12 01:36:48,021 - DEBUG - [10.0.0.38]: Running cmd : docker exec --privileged -it contrail-kubernetes-master-kubemanager-1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2025-08-12 01:36:48,157 - DEBUG - Output : cluster_project={}
2025-08-12 01:36:58,285 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:36:58,319 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:36:58,392 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:37:03,430 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:37:03,505 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:37:03,581 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:37:03,617 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:37:03,692 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2025-08-12 01:37:03,920 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-12 01:37:06,480 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_ApplicationPolicySetReq?
2025-08-12 01:37:06,486 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AclReq?
2025-08-12 01:37:06,513 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AclReq?
2025-08-12 01:37:08,541 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AclReq?
2025-08-12 01:40:16,551 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_AgentXmppConnectionStatusReq?
2025-08-12 01:40:16,583 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=04bc01b2-cb63-4ea0-a98c-92245d66ad6c
2025-08-12 01:40:16,599 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:16,605 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:16,611 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:16,614 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:16,618 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=7bc7d613-c0d2-49ae-b9f4-574a98593467
2025-08-12 01:40:16,621 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f373f0d8-050f-4f9b-877c-e49e7487fa7f
2025-08-12 01:40:16,625 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=d2be86e9-a449-4f31-bc1b-09b3f7a220e6
2025-08-12 01:40:21,658 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=d2be86e9-a449-4f31-bc1b-09b3f7a220e6
2025-08-12 01:40:21,659 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=7bc7d613-c0d2-49ae-b9f4-574a98593467
2025-08-12 01:40:21,661 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:21,662 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:21,664 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:21,665 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:21,667 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=04bc01b2-cb63-4ea0-a98c-92245d66ad6c
2025-08-12 01:40:21,669 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=f373f0d8-050f-4f9b-877c-e49e7487fa7f
2025-08-12 01:40:21,693 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cff578be-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:21,695 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:cdf2b1b2-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:21,698 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d065eeb4-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:21,700 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ccdd262c-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:26,693 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:26,694 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:26,696 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:26,698 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:31,710 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:31,712 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:31,713 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:31,714 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:36,728 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:36,730 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:36,733 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:36,734 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:41,749 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:41,750 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:41,752 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:41,753 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:46,767 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:46,769 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:46,770 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:46,771 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:51,839 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=eaba0f12-c404-4814-a60c-1b05de6e6e7c
2025-08-12 01:40:51,840 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=9a1207c0-0545-4e86-8186-4529668381ca
2025-08-12 01:40:51,841 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=653172c6-541e-494b-98af-d7e6b171d84b
2025-08-12 01:40:51,842 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_VmListReq?uuid=be440c65-a1e7-420b-8b81-e8b029bec0b2
2025-08-12 01:40:51,855 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d15669a2-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:51,857 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d0e4db16-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:51,859 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d247c6da-771c-11f0-bea2-fa163e5be5c4
2025-08-12 01:40:51,860 - DEBUG - Requesting: http://10.0.0.249:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:d1daf88e-771c-11f0-bea2-fa163e5be5c4