2024-11-15 01:21:14,912 - DEBUG - [10.0.0.47]: Running cmd : hostname
2024-11-15 01:21:15,214 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-802-1
2024-11-15 01:21:15,214 - DEBUG - [10.0.0.47]: Running cmd : hostname -f
2024-11-15 01:21:15,248 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-802-1
2024-11-15 01:21:15,248 - DEBUG - [10.0.0.47]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2024-11-15 01:21:15,325 - DEBUG - Output : NAMES
contrail_test_3s60VPpUa
webui_web_1
webui_job_1
contrail-kubernetes-master_kubemanager_1
control_named_1
control_control_1
control_dns_1
control_nodemgr_1
config_database_zookeeper_1
config_database_nodemgr_1
config_database_rabbitmq_1
config_database_cassandra_1
config_api_devicemgr_1
config_api_svcmonitor_1
config_api_nodemgr_1
config_api_dnsmasq_1
config_api_api_1
config_api_schema_1
analytics_database_cassandra_1
analytics_database_nodemgr_1
analytics_database_query-engine_1
redis_redis_1
analytics_snmp_snmp-collector_1
analytics_snmp_topology_1
analytics_snmp_nodemgr_1
analytics_alarm_kafka_1
analytics_alarm_nodemgr_1
analytics_alarm_alarm-gen_1
analytics_nodemgr_1
analytics_collector_1
analytics_api_1
2024-11-15 01:21:15,326 - DEBUG - [10.0.0.47]: Running cmd : ip addr show | grep 'inet .*/.* brd ' | awk '{print $2}'
2024-11-15 01:21:15,405 - DEBUG - Output : 10.0.0.47/24
172.17.0.1/16
2024-11-15 01:21:15,406 - DEBUG - [10.0.0.47]: Running cmd : getent hosts 10.0.0.47 | head -n 1 | awk '{print $2}'
2024-11-15 01:21:15,450 - DEBUG - Output : cn-jenkins-deploy-platform-juju-k8s-802-1
2024-11-15 01:21:15,549 - DEBUG - [10.0.0.40]: Running cmd : hostname
2024-11-15 01:21:15,784 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-802-1
2024-11-15 01:21:15,785 - DEBUG - [10.0.0.40]: Running cmd : hostname -f
2024-11-15 01:21:15,853 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-802-1
2024-11-15 01:21:15,853 - DEBUG - [10.0.0.40]: Running cmd : docker ps -a 2>/dev/null | grep -v "/pause\|/usr/bin/pod\|nova_api_\|contrail.*init\|init.*contrail\|provisioner\|placement" | awk '{print $NF}'
2024-11-15 01:21:15,908 - DEBUG - Output : NAMES
vrouter_nodemgr_1
vrouter_vrouter-agent_1
2024-11-15 01:21:15,908 - DEBUG - [10.0.0.40]: Running cmd : ip addr show dev vhost0 | grep 'inet .*/.* brd ' | awk '{print $2}'
2024-11-15 01:21:15,977 - DEBUG - Output : 10.0.0.40/24
2024-11-15 01:21:15,977 - DEBUG - [10.0.0.40]: Running cmd : getent hosts 10.0.0.40 | head -n 1 | awk '{print $2}'
2024-11-15 01:21:16,011 - DEBUG - Output : an-jenkins-deploy-platform-juju-k8s-802-1
2024-11-15 01:21:16,104 - INFO - Using existing project ['default-domain', 'k8s-default'](16a0499e-c99d-42f4-9e47-811296949bad)
2024-11-15 01:21:16,220 - INFO - Using existing project ['default-domain', 'k8s-default'](16a0499e-c99d-42f4-9e47-811296949bad)
2024-11-15 01:21:16,384 - INFO - ================================================================================
2024-11-15 01:21:16,384 - INFO - STARTING TEST : test_ingress_ip_assignment
2024-11-15 01:21:16,385 - INFO - TEST DESCRIPTION :
Verify that Ingress gets a CLuster IP which is reachable to Pods in same
namespace. Also verify that a Floating IP is assigned to the Ingress
from the Public FIP poo.
Steps:
1. Create a service with 2 pods running nginx
2. Create an ingress out of this service
3. From another Pod do a wget on the ingress Cluster ip
Validate that Ingress get a IP from Public FIP pool which might/might not be accessible.
Validate that service and its loadbalancing work
2024-11-15 01:21:16,579 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-15 01:21:16,587 - DEBUG - Nothing to compare xmpp stats {'10.0.0.40': {'10.0.0.47': '0'}} with
2024-11-15 01:21:16,587 - INFO - Initial checks done. Running the testcase now
2024-11-15 01:21:16,587 - INFO -
2024-11-15 01:21:16,591 - INFO - Namespace default already exists
2024-11-15 01:21:16,592 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2024-11-15 01:21:16,697 - DEBUG - Output : cluster_project={}
2024-11-15 01:21:16,753 - INFO - Project default-domain:k8s-default for Namespace default is seen in contrail-api
2024-11-15 01:21:16,754 - DEBUG - Project uuid in Contrail is 16a0499e-c99d-42f4-9e47-811296949bad
2024-11-15 01:21:16,754 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2024-11-15 01:21:16,882 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2024-11-15 01:21:16,882 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_NamespaceDatabaseList?namespace_uuid=a7328c41-e59c-4c4b-9c21-f0425addfd38
2024-11-15 01:21:16,887 - INFO - Namespace default with uuid a7328c41-e59c-4c4b-9c21-f0425addfd38 found in kube manager
2024-11-15 01:21:16,887 - INFO - Namespace default verification passed
2024-11-15 01:21:16,890 - DEBUG - Service ctest-nginx-svc-50500314 not present
2024-11-15 01:21:16,890 - INFO - Creating service ctest-nginx-svc-50500314
2024-11-15 01:21:16,917 - INFO - Created Service ctest-nginx-svc-50500314
2024-11-15 01:21:16,928 - DEBUG - Pod ctest-nginx-pod-75734645 not present
2024-11-15 01:21:16,928 - INFO - Creating Pod ctest-nginx-pod-75734645
2024-11-15 01:21:16,968 - DEBUG - Pod : ctest-nginx-pod-75734645 UUID is 6d402896-060a-443f-8a02-3dd7fadd2ce9
2024-11-15 01:21:16,974 - DEBUG - Pod ctest-nginx-pod-58456245 not present
2024-11-15 01:21:16,977 - INFO - Creating Pod ctest-nginx-pod-58456245
2024-11-15 01:21:17,019 - DEBUG - Pod : ctest-nginx-pod-58456245 UUID is ada0e6e1-fb3e-417a-b309-126b6598a7b6
2024-11-15 01:21:17,289 - INFO - Created VN __public__, UUID :d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:17,586 - DEBUG - Requesting: http://10.0.0.47:8082/domains
2024-11-15 01:21:17,595 - DEBUG - Requesting: http://10.0.0.47:8082/domain/710ad691-1037-4de1-b7e3-cacf956b3942
2024-11-15 01:21:17,625 - DEBUG - Requesting: http://10.0.0.47:8082/project/16a0499e-c99d-42f4-9e47-811296949bad
2024-11-15 01:21:17,676 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:17,731 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:17,754 - DEBUG - Requesting: http://10.0.0.47:8082/routing-instance/fbf6c0e3-0b63-4daa-86d6-f2f64e4a101d
2024-11-15 01:21:17,774 - DEBUG - Requesting: http://10.0.0.47:8082/routing-instance/fbf6c0e3-0b63-4daa-86d6-f2f64e4a101d
2024-11-15 01:21:17,784 - DEBUG - Requesting: http://10.0.0.47:8082/route-target/57a53bb4-a0f1-4bfd-993f-4648af4b54f2
2024-11-15 01:21:17,797 - DEBUG - Route Targets: ['target:64512:8000007']
2024-11-15 01:21:17,797 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:17,821 - DEBUG - Requesting: http://10.0.0.47:8082/routing-instance/fbf6c0e3-0b63-4daa-86d6-f2f64e4a101d
2024-11-15 01:21:17,873 - INFO - Verified VN network id 13 for VN d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:17,873 - INFO - Verifications in API Server for VN __public__ passed
2024-11-15 01:21:17,873 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:17,897 - DEBUG - Requesting: http://10.0.0.47:8082/routing-instance/fbf6c0e3-0b63-4daa-86d6-f2f64e4a101d
2024-11-15 01:21:17,920 - DEBUG - Requesting: http://10.0.0.47:8082/routing-instance/fbf6c0e3-0b63-4daa-86d6-f2f64e4a101d
2024-11-15 01:21:17,944 - DEBUG - Requesting: http://10.0.0.47:8082/route-target/57a53bb4-a0f1-4bfd-993f-4648af4b54f2
2024-11-15 01:21:17,958 - DEBUG - Requesting: http://10.0.0.47:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2024-11-15 01:21:17,966 - DEBUG - Control-node 10.0.0.47 : VN object is : {'node_name': 'virtual-network:default-domain:k8s-default:__public__', 'interests': '-', 'advertised': '-', 'dbentryflags': None, 'obj_info': {'sequence_number': '0', 'origin': 'Cassandra', 'data': {'id-perms': {'permissions': {'owner': 'cloud-admin', 'owner-access': '7', 'group': 'cloud-admin-group', 'group-access': '7', 'other-access': '7'}, 'uuid': {'uuid-mslong': '15378702074815923468', 'uuid-lslong': '11865821177872375588'}, 'enable': 'true', 'created': '2024-11-15T01:21:17', 'last-modified': '2024-11-15T01:21:17', 'description': None, 'user-visible': 'true', 'creator': None}}}, 'neighbors': None, 'last_modified': '00:00:00.498902'}
2024-11-15 01:21:17,966 - DEBUG - Requesting: http://10.0.0.47:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:21:17,975 - DEBUG - Route Targets: ['target:64512:8000007']
2024-11-15 01:21:17,975 - INFO - On all control nodes, Config, RI and RT verification for VN __public__ passed
2024-11-15 01:21:17,976 - DEBUG - ====Verifying policy data for __public__ in API_Server ======
2024-11-15 01:21:17,976 - DEBUG - Requesting: http://10.0.0.47:8082/domains
2024-11-15 01:21:17,983 - DEBUG - Requesting: http://10.0.0.47:8082/domain/710ad691-1037-4de1-b7e3-cacf956b3942
2024-11-15 01:21:18,010 - DEBUG - Requesting: http://10.0.0.47:8082/project/16a0499e-c99d-42f4-9e47-811296949bad
2024-11-15 01:21:18,089 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:18,104 - DEBUG - =>VN __public__ has no policy to be verified
2024-11-15 01:21:18,104 - DEBUG - Verifying the vn in opserver
2024-11-15 01:21:18,104 - DEBUG - Verifying the default-domain:k8s-default:__public__ virtual network link through opserver 10.0.0.47
2024-11-15 01:21:18,105 - DEBUG - Requesting: http://10.0.0.47:8081/analytics/uves/virtual-networks
2024-11-15 01:21:18,111 - DEBUG - vn link and name as {'name': 'default-domain:k8s-default:__public__', 'href': 'http://10.0.0.47:8081/analytics/uves/virtual-network/default-domain:k8s-default:__public__?flat'}
2024-11-15 01:21:18,111 - INFO - Validated that VN default-domain:k8s-default:__public__ is found in opserver
2024-11-15 01:21:18,111 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:21:18,121 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:21:18,131 - DEBUG - Do not have enough data to verify VN in agent
2024-11-15 01:21:18,131 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:21:18,143 - DEBUG - VRF ids for VN __public__: {}
2024-11-15 01:21:18,273 - INFO - Creating Floating IP pool __fip_pool_public__ in API Server
2024-11-15 01:21:18,406 - DEBUG - Requesting: http://10.0.0.47:8082/domains
2024-11-15 01:21:18,412 - DEBUG - Requesting: http://10.0.0.47:8082/domain/710ad691-1037-4de1-b7e3-cacf956b3942
2024-11-15 01:21:18,422 - DEBUG - Requesting: http://10.0.0.47:8082/project/16a0499e-c99d-42f4-9e47-811296949bad
2024-11-15 01:21:18,522 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:18,537 - DEBUG - Requesting: http://10.0.0.47:8082/floating-ip-pool/b6ade2ca-e917-460b-80e4-63ae997a8fa3
2024-11-15 01:21:18,544 - DEBUG - Requesting: http://10.0.0.47:8082/domains
2024-11-15 01:21:18,549 - DEBUG - Requesting: http://10.0.0.47:8082/domain/710ad691-1037-4de1-b7e3-cacf956b3942
2024-11-15 01:21:18,558 - DEBUG - Requesting: http://10.0.0.47:8082/project/16a0499e-c99d-42f4-9e47-811296949bad
2024-11-15 01:21:18,647 - DEBUG - Requesting: http://10.0.0.47:8082/virtual-network/d56c200e-a3f7-450c-a4ab-dd7e5363df24
2024-11-15 01:21:18,661 - DEBUG - FIP Pool __fip_pool_public__ found in API Server
2024-11-15 01:21:18,661 - INFO - Verification for FIP pool __fip_pool_public__ in API Server passed
2024-11-15 01:21:18,676 - DEBUG - Requesting: http://10.0.0.47:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2024-11-15 01:21:18,682 - DEBUG - Control-node Ifmap-view has FIP pool __fip_pool_public__ information
2024-11-15 01:21:18,682 - INFO - Verification for FIP pool __fip_pool_public__ in Control-node passed
2024-11-15 01:21:18,686 - DEBUG - Ingress ctest-nginx-ingress-20042265 not present
2024-11-15 01:21:18,686 - INFO - Creating Ingress ctest-nginx-ingress-20042265
2024-11-15 01:21:18,694 - INFO - Created Ingress ctest-nginx-ingress-20042265
2024-11-15 01:21:18,697 - DEBUG - Cluster IP not yet seen for Ingress ctest-nginx-ingress-20042265
2024-11-15 01:21:21,702 - DEBUG - For Ingress ctest-nginx-ingress-20042265, Cluster IP: 10.47.255.249, External IPs ['65.121.123.60']
2024-11-15 01:21:21,702 - INFO - Verifications in k8s passed for Ingress ctest-nginx-ingress-20042265
2024-11-15 01:21:21,702 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_LoadbalancerDatabaseList?x=97eb6bce-d080-4c79-8321-79d91cc297dd
2024-11-15 01:21:21,708 - INFO - Ingress ctest-nginx-ingress-20042265 with uuid 97eb6bce-d080-4c79-8321-79d91cc297dd found in kube manager
2024-11-15 01:21:21,708 - INFO - Ingress ctest-nginx-ingress-20042265 verification passed
2024-11-15 01:21:21,710 - DEBUG - Pod ctest-busybox-pod-65027392 not present
2024-11-15 01:21:21,710 - INFO - Creating Pod ctest-busybox-pod-65027392
2024-11-15 01:21:21,738 - DEBUG - Pod : ctest-busybox-pod-65027392 UUID is 23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:21,743 - DEBUG - Pod ctest-nginx-pod-75734645 not in running state.Currently in Pending
2024-11-15 01:21:26,751 - DEBUG - Pod ctest-nginx-pod-75734645 not in running state.Currently in Pending
2024-11-15 01:21:31,757 - INFO - Pod ctest-nginx-pod-75734645 is in running state.Got IP 10.47.255.251
2024-11-15 01:21:31,803 - DEBUG - Pod ctest-nginx-pod-75734645 has vmi eaa1aee6-a2ef-11ef-8635-fa163eb8ee38
2024-11-15 01:21:31,803 - INFO - Verified pod ctest-nginx-pod-75734645 in contrail-api
2024-11-15 01:21:31,803 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:21:31,821 - DEBUG - VMI eaa1aee6-a2ef-11ef-8635-fa163eb8ee38 is active in agent 10.0.0.40
2024-11-15 01:21:31,821 - INFO - Verified Pod ctest-nginx-pod-75734645 in agent 10.0.0.40
2024-11-15 01:21:31,821 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_VirtualMachineDatabaseList?x=6d402896-060a-443f-8a02-3dd7fadd2ce9
2024-11-15 01:21:31,830 - INFO - Pod ctest-nginx-pod-75734645 with uuid 6d402896-060a-443f-8a02-3dd7fadd2ce9 found in kube manager
2024-11-15 01:21:31,830 - INFO - Pod ctest-nginx-pod-75734645 verification passed
2024-11-15 01:21:31,904 - DEBUG - [Pod ctest-nginx-pod-75734645] Cmd: echo ctest-nginx-pod-75734645 > /usr/share/nginx/html/index.html, Output:
2024-11-15 01:21:31,910 - INFO - Pod ctest-nginx-pod-58456245 is in running state.Got IP 10.47.255.250
2024-11-15 01:21:31,973 - DEBUG - Pod ctest-nginx-pod-58456245 has vmi eb11d68a-a2ef-11ef-8635-fa163eb8ee38
2024-11-15 01:21:31,974 - INFO - Verified pod ctest-nginx-pod-58456245 in contrail-api
2024-11-15 01:21:31,974 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:21:31,992 - DEBUG - VMI eb11d68a-a2ef-11ef-8635-fa163eb8ee38 is active in agent 10.0.0.40
2024-11-15 01:21:31,992 - INFO - Verified Pod ctest-nginx-pod-58456245 in agent 10.0.0.40
2024-11-15 01:21:31,992 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_VirtualMachineDatabaseList?x=ada0e6e1-fb3e-417a-b309-126b6598a7b6
2024-11-15 01:21:31,997 - INFO - Pod ctest-nginx-pod-58456245 with uuid ada0e6e1-fb3e-417a-b309-126b6598a7b6 found in kube manager
2024-11-15 01:21:31,997 - INFO - Pod ctest-nginx-pod-58456245 verification passed
2024-11-15 01:21:32,068 - DEBUG - [Pod ctest-nginx-pod-58456245] Cmd: echo ctest-nginx-pod-58456245 > /usr/share/nginx/html/index.html, Output:
2024-11-15 01:21:32,074 - DEBUG - Pod ctest-busybox-pod-65027392 not in running state.Currently in Pending
2024-11-15 01:21:37,082 - INFO - Pod ctest-busybox-pod-65027392 is in running state.Got IP 10.47.255.248
2024-11-15 01:21:37,105 - DEBUG - Pod ctest-busybox-pod-65027392 has vmi ed80241c-a2ef-11ef-8635-fa163eb8ee38
2024-11-15 01:21:37,105 - INFO - Verified pod ctest-busybox-pod-65027392 in contrail-api
2024-11-15 01:21:37,105 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:21:37,139 - DEBUG - VMI ed80241c-a2ef-11ef-8635-fa163eb8ee38 is active in agent 10.0.0.40
2024-11-15 01:21:37,140 - INFO - Verified Pod ctest-busybox-pod-65027392 in agent 10.0.0.40
2024-11-15 01:21:37,140 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_VirtualMachineDatabaseList?x=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:37,149 - INFO - Pod ctest-busybox-pod-65027392 with uuid 23523117-2b95-4eb7-ada3-a538aabc5282 found in kube manager
2024-11-15 01:21:37,149 - INFO - Pod ctest-busybox-pod-65027392 verification passed
2024-11-15 01:21:37,217 - DEBUG - [Pod ctest-busybox-pod-65027392] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: ctest-nginx-pod-75734645
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2024-11-15 01:21:37,217 - DEBUG - [Pod Pod ctest-busybox-pod-65027392] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2024-11-15 01:21:37,218 - DEBUG - [Pod Pod ctest-busybox-pod-65027392] Cmd output: ctest-nginx-pod-75734645
Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
2024-11-15 01:21:37,285 - DEBUG - [Pod ctest-busybox-pod-65027392] Cmd: wget http://10.47.255.249:80/ -O - -T 5 -t 1 , Output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-58456245
2024-11-15 01:21:37,285 - DEBUG - [Pod Pod ctest-busybox-pod-65027392] Cmd wget http://10.47.255.249:80/ -O - -T 5 -t 1 passed
2024-11-15 01:21:37,285 - DEBUG - [Pod Pod ctest-busybox-pod-65027392] Cmd output: Connecting to 10.47.255.249:80 (10.47.255.249:80)
writing to stdout
- 100% |********************************| 25 0:00:00 ETA
written to stdout
ctest-nginx-pod-58456245
2024-11-15 01:21:37,285 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-75734645': 1, 'ctest-nginx-pod-58456245': 1}
2024-11-15 01:21:37,285 - WARNING - Nothing to delete parallely
2024-11-15 01:21:37,285 - INFO - Deleting pod default:ctest-busybox-pod-65027392
2024-11-15 01:21:37,300 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:37,317 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:21:42,318 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:42,325 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:21:47,326 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:47,336 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:21:52,337 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:52,344 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:21:57,345 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:21:57,353 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:22:02,354 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:22:02,361 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:22:07,361 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:22:07,369 - WARNING - Pod uuid 23523117-2b95-4eb7-ada3-a538aabc5282 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:22:12,369 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=23523117-2b95-4eb7-ada3-a538aabc5282
2024-11-15 01:22:12,378 - DEBUG - Pod 23523117-2b95-4eb7-ada3-a538aabc5282 is not in agent 10.0.0.40 VM list
2024-11-15 01:22:12,378 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:ed80241c-a2ef-11ef-8635-fa163eb8ee38
2024-11-15 01:22:12,384 - DEBUG - VMI ed80241c-a2ef-11ef-8635-fa163eb8ee38 is removed from agent 10.0.0.40
2024-11-15 01:22:12,384 - INFO - Verified that pod ctest-busybox-pod-65027392 is removed in agent
2024-11-15 01:22:12,384 - INFO - Deleting Ingress : ctest-nginx-ingress-20042265
2024-11-15 01:22:12,391 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_LoadbalancerDatabaseList?x=97eb6bce-d080-4c79-8321-79d91cc297dd
2024-11-15 01:22:12,398 - ERROR - Ingress ctest-nginx-ingress-20042265 with uuid 97eb6bce-d080-4c79-8321-79d91cc297dd still found in kube manager
2024-11-15 01:22:13,404 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_LoadbalancerDatabaseList?x=97eb6bce-d080-4c79-8321-79d91cc297dd
2024-11-15 01:22:13,409 - ERROR - Ingress ctest-nginx-ingress-20042265 with uuid 97eb6bce-d080-4c79-8321-79d91cc297dd still found in kube manager
2024-11-15 01:22:14,410 - DEBUG - Requesting: http://10.0.0.47:8108/Snh_LoadbalancerDatabaseList?x=97eb6bce-d080-4c79-8321-79d91cc297dd
2024-11-15 01:22:14,415 - INFO - Ingress ctest-nginx-ingress-20042265 with uuid 97eb6bce-d080-4c79-8321-79d91cc297dd deleted successfully from kube manager
2024-11-15 01:22:14,415 - INFO - Verifications on Ingress %s cleanup passed
2024-11-15 01:22:14,415 - INFO - Deleting the FIP pool __fip_pool_public__
2024-11-15 01:22:14,471 - DEBUG - Requesting: http://10.0.0.47:8083/Snh_IFMapTableShowReq?table_name=floating-ip-pool&search_string=floating-ip-pool:default-domain:k8s-default:__public__:__fip_pool_public__
2024-11-15 01:22:14,483 - DEBUG - Control-node Ifmap-view does not have FIP pool __fip_pool_public__ information
2024-11-15 01:22:14,483 - INFO - Deleting VN __public__
2024-11-15 01:22:14,591 - DEBUG - Requesting: http://10.0.0.47:8082/routing-instance/fbf6c0e3-0b63-4daa-86d6-f2f64e4a101d
2024-11-15 01:22:14,600 - DEBUG - Response Code: 404
2024-11-15 01:22:14,600 - DEBUG - Requesting: http://10.0.0.47:8082/domains
2024-11-15 01:22:14,623 - DEBUG - Requesting: http://10.0.0.47:8082/domain/710ad691-1037-4de1-b7e3-cacf956b3942
2024-11-15 01:22:14,635 - DEBUG - Requesting: http://10.0.0.47:8082/project/16a0499e-c99d-42f4-9e47-811296949bad
2024-11-15 01:22:14,684 - INFO - Validated that VN __public__ is not found in API Server
2024-11-15 01:22:14,684 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VnListReq?name=default-domain:k8s-default:__public__
2024-11-15 01:22:14,692 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:22:14,698 - DEBUG - VN __public__ is not present in Agent 10.0.0.40
2024-11-15 01:22:14,698 - INFO - Validated that VN __public__ is not in any agent
2024-11-15 01:22:14,698 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VrfListReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:22:14,705 - DEBUG - Requesting: http://10.0.0.47:8083/Snh_ShowRoutingInstanceReq?name=default-domain:k8s-default:__public__:__public__
2024-11-15 01:22:14,712 - DEBUG - Requesting: http://10.0.0.47:8083/Snh_IFMapTableShowReq?table_name=virtual-network&search_string=virtual-network:default-domain:k8s-default:__public__
2024-11-15 01:22:14,717 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN __public__ info
2024-11-15 01:22:14,718 - INFO - Deleting pod default:ctest-nginx-pod-58456245
2024-11-15 01:22:14,729 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=ada0e6e1-fb3e-417a-b309-126b6598a7b6
2024-11-15 01:22:14,739 - WARNING - Pod uuid ada0e6e1-fb3e-417a-b309-126b6598a7b6 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:22:19,740 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=ada0e6e1-fb3e-417a-b309-126b6598a7b6
2024-11-15 01:22:19,748 - DEBUG - Pod ada0e6e1-fb3e-417a-b309-126b6598a7b6 is not in agent 10.0.0.40 VM list
2024-11-15 01:22:19,748 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:eb11d68a-a2ef-11ef-8635-fa163eb8ee38
2024-11-15 01:22:19,754 - DEBUG - VMI eb11d68a-a2ef-11ef-8635-fa163eb8ee38 is removed from agent 10.0.0.40
2024-11-15 01:22:19,755 - INFO - Verified that pod ctest-nginx-pod-58456245 is removed in agent
2024-11-15 01:22:19,755 - INFO - Deleting pod default:ctest-nginx-pod-75734645
2024-11-15 01:22:19,769 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=6d402896-060a-443f-8a02-3dd7fadd2ce9
2024-11-15 01:22:19,790 - WARNING - Pod uuid 6d402896-060a-443f-8a02-3dd7fadd2ce9 is still seen in agent 10.0.0.40 VM list
2024-11-15 01:22:24,791 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=6d402896-060a-443f-8a02-3dd7fadd2ce9
2024-11-15 01:22:24,799 - DEBUG - Pod 6d402896-060a-443f-8a02-3dd7fadd2ce9 is not in agent 10.0.0.40 VM list
2024-11-15 01:22:24,799 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:eaa1aee6-a2ef-11ef-8635-fa163eb8ee38
2024-11-15 01:22:24,805 - DEBUG - VMI eaa1aee6-a2ef-11ef-8635-fa163eb8ee38 is removed from agent 10.0.0.40
2024-11-15 01:22:24,805 - INFO - Verified that pod ctest-nginx-pod-75734645 is removed in agent
2024-11-15 01:22:24,805 - INFO - Deleting service : ctest-nginx-svc-50500314
2024-11-15 01:22:24,958 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-15 01:22:24,966 - DEBUG - No XMPP flaps were noticed during the test
2024-11-15 01:22:24,966 - INFO - END TEST : test_ingress_ip_assignment : PASSED[0:01:08]
2024-11-15 01:22:24,966 - INFO - --------------------------------------------------------------------------------
2024-11-15 01:22:31,415 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2024-11-15 01:22:31,607 - DEBUG - Output : cluster_project={}
2024-11-15 01:22:31,689 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'netstat -antp | grep :8108 | grep LISTEN'
2024-11-15 01:22:31,859 - DEBUG - Output : tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 1/python3
2024-11-15 01:22:31,891 - DEBUG - [10.0.0.47]: Running cmd : docker exec --privileged -it contrail-kubernetes-master_kubemanager_1 /bin/bash -c 'grep "^[ ]*cluster_project" /etc/contrail/contrail-kubernetes.conf'
2024-11-15 01:22:32,050 - DEBUG - Output : cluster_project={}
2024-11-15 01:22:47,230 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:47,275 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:47,362 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:47,462 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:52,535 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:52,626 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:52,671 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:52,775 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,
2024-11-15 01:22:53,049 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-15 01:22:55,895 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_ApplicationPolicySetReq?
2024-11-15 01:22:55,902 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AclReq?
2024-11-15 01:22:55,933 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AclReq?
2024-11-15 01:22:57,980 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AclReq?
2024-11-15 01:26:06,096 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_AgentXmppConnectionStatusReq?
2024-11-15 01:26:06,175 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=5af10c86-9a51-4c0b-b120-893302f6dca8
2024-11-15 01:26:06,233 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:06,277 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=36d3a14b-ad9d-4c97-9d04-1b2df6ceae6e
2024-11-15 01:26:06,306 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:06,311 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=2ac9514b-8947-45e1-be1a-84ac0bbeca1a
2024-11-15 01:26:06,340 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:06,351 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=633375d7-57c6-4dc5-8275-3be3cd82ea7d
2024-11-15 01:26:06,362 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:11,379 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=5af10c86-9a51-4c0b-b120-893302f6dca8
2024-11-15 01:26:11,386 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1719d912-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:11,388 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:11,472 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=633375d7-57c6-4dc5-8275-3be3cd82ea7d
2024-11-15 01:26:11,474 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:11,475 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:11,476 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=2ac9514b-8947-45e1-be1a-84ac0bbeca1a
2024-11-15 01:26:11,477 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=36d3a14b-ad9d-4c97-9d04-1b2df6ceae6e
2024-11-15 01:26:11,480 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:11,500 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:18e3e486-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:11,502 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:185f5cac-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:11,503 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:17ad51f6-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:16,398 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:16,498 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:16,500 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:16,501 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:21,406 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:21,513 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:21,515 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:21,516 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:26,414 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:26,529 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:26,531 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:26,533 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:31,423 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:31,545 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:31,547 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:31,548 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:36,434 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:36,564 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:36,566 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:36,567 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:41,478 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=9bb82dc2-2168-44fe-a2a8-928d7b0f32c5
2024-11-15 01:26:41,486 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1a8528cc-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:41,599 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=21122fc7-e888-48a7-be54-18cd5f7b68e4
2024-11-15 01:26:41,601 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=e8719320-7674-41de-94e2-860fb3003360
2024-11-15 01:26:41,603 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_VmListReq?uuid=78963838-93f1-4d54-a156-03f70239ca7a
2024-11-15 01:26:41,615 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:1b03f7d8-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:41,618 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:19ff6610-a2f0-11ef-8635-fa163eb8ee38
2024-11-15 01:26:41,619 - DEBUG - Requesting: http://10.0.0.40:8085/Snh_PageReq?x=begin:-1,end:-1,table:db.interface.0,uuid:19792ca8-a2f0-11ef-8635-fa163eb8ee38